Talk:WorkingWiki

From projects

Jump to: navigation, search

Contents

WorkingWiki Bug Reports

If you have a bug to report, please add it to the workingwiki bug tracker at SourceForge.net. First search for related items there, then use "Add new" followed by "Add artifact". If the bug tracker isn't working for you, then just send your report to the WW developer at (worden.lee at gmail dot com). See WorkingWiki/Documentation#Bug tracking for guidelines for bug reports.


If the system's behavior has recently changed in a puzzling way, you might want to look over the Change Log or the recent changes to the source code.

Old bug reports that need to be reviewed, transferred to SourceForge

Special:ManageProject and directory listing page

  • If you fill in the 'page' before the 'source filename', it shouldn't replace it with the default vaue
  • Should have option to remove a wiki page when removing a source-file leaves it blank
  • Should be able to remove a whole project and some or all of its files
  • Also rename a project and convert the tags on wiki pages to use the new name.
    • Maybe 'delete' and 'move' tabs atop the page
  • Warn when the project name recorded in a project-description doesn't match its location.
  • Should have 'create new project' feature, maybe (though not on this page, exactly)
    • There are already ways to create a project, of course: by creating a source-file that belongs to it, by importing, and by typing its name into the project search bar.
  • parse a tex file for \include, \includegraphics etc when it's added via this page, and offer to add those files to the project as well.
  • quick-links to edit a page, and to download a file (using Special:GetProject?display=raw)
  • remove a file without jumping to the top of the WD page (this will probably have to wait until I get into writing AJAX code, which is not a top priority so far)

Special:GetProjectFile

  • Consider http://www.mediawiki.org/wiki/Extension:AllowGetParamsInWikilinks as an alternative to the /filename=X/project=P workaround.
    • This would also allow internal links to resources/ files, which don't currently support the /filename=X workaround.
  • A possible change: always display the file in the wiki page rather than offering the file for download, unless display=raw is explicit? (Currently it does the download by default for images and filetypes that can't be displayed.)

Import and Export

Export
  • Export should (as an option?) include default-rules.makefile and whatever else is needed to do the compilation that happens on the wiki. Maybe just include the whole resources/ directory?
  • Export should (as an option) include the wikitext of the pages that contain the source files, so the whole thing can be reconstructed by importing to a different wiki.
  • Am adding these 2, but have not deployed to users yet.
ImportProjectFiles
  • Need to check everything for injection possibilities.
  • bad error message when file is too big:
    • A database query syntax error has occurred. This may indicate a bug in the software. . . . MySQL returned error "1153: Got a packet bigger than 'max_allowed_packet' bytes (localhost)".
  • When project name is bad, should warn, not just proceed with "(invalid project name)".
  • When unpacking uploaded .tar.gz files: Wikipedia says: GNU tar by default refuses to create or extract absolute paths, but is still vulnerable to parent-directory references. So I would need to check tar contents for ../ before extracting. In practice it seems to be catching this case — but I should trap it explicitly anyway?
  • is there much danger of overrunning the php posting limits when uploading multiple files?
    • I'm limiting it to 20 files at a time right now — is it too many in some cases?
  • This Special: page code is based on a customized version of the Special:Upload code from MW 1.15, with a bunch of changes — functions split into many parts so they can be called separately and overloaded, and backward compatibility for lalashan's wikis, which use MW 1.13. Mediawiki is preparing to switch to the new-upload branch under development which will provide a substantially different Special:Upload with its own refactoring of the functions. My SpecialCustomUpload class should be rewritten to use big chunks of the new-upload code, so it'll be mostly compatible when that change happens. (It's happened — will be in MW 1.16 –lw.) This is becoming especially important as the code we've got is losing compatibility with new revisions of MW.
  • uploading to a File: page that's been deleted doesn't work right in importProjectFiles.php. Is it an issue in Special:ImportProjectFiles as well? Fix it.

HTML / XHTML

  • When to serve which:

WW gives regular HTML pages to most viewers, and XHTML with MathML to people who are logged in and have selected the MathML option in their preferences.

We have a site notice at the top of the page when a user could be seeing MathML but isn't. Unregistered viewers don't get MathML, but there is a screenshot on the projects wiki to show them what it looks like.

To do: instead of changing to Content-type: application/xhtml+xml anytime the user and browser are compatible, better to do it only on pages that actually have mathml content? that would save redundant parser cache entries, and cut down on incompatibilities with other extensions.

To do: make it work on Internet Explorer/MathPlayer. It's not working currently.

To do: include an option to install WW without MathML support. This would ideally allow WW to work on a MW installation that isn't patched (this is not a need, now that it doesn't require patches any more).

To come: MediaWiki is likely to switch from outputting XHTML-compliant HTML to using HTML5. HTML5 supports MathML so we'll be fine, but eventually will have to make WW compatible with the HTML5 option and switch our wikis over.

  • inline math equations don't have the blue hover color when they're display-style (i.e. taking up a full line). Why not?

LaTeX

  • Rotated image that isn't rotated. See Stochasticity_Chapter. (this is an old bug. is it still an issue?)
  • see WorkingWiki/LaTeXML Notes
  • Sometimes wordy figure captions make the figure too wide, even the width of the whole column. This is true both in latex figures and in hand-done <div> style figures.
  • Revised issue: Neither method of using pdfs for images seems to work, currently. (neither setting LATEX=pdflatex in the Makefile or setting pdflink=X.pdflatex.pdf) Either method gives the same result: in the log, what turns up is 
Warning: Missing graphic for <graphics graphic="2patch.pdf" options="width=505.89pt" xml:id="S0.F1.g1" fragid="S0.F1.g1"/>; skipping

and an empty graphic region shows up in the document. For an example, go to http://ibis.ucdavis.edu/mw/index.php/DebugPage, and if necessary, change the suffix of the image to .pdf. Note: this is not a big deal.

  • Without MathML, \usepackage{amsmath} and the split environment do not work together: equations that use split get rendered as 6x6 blank images. Using MathML fixes the problem. An example is at http://ibis.ucdavis.edu/mw/index.php/DebugPage .

Makefiles

  • Is there a way to make the site makefile and/or default makefile available for editing as a wiki page? In which wiki, since they apply to the whole wiki farm? Also, the default makefile is part of the svn repository — if it's been edited in the wiki and there are new changes in the repository how will it be updated? Should there also be a per-wiki makefile that can be edited in the wiki?
    • I think the answer is to create a project to contain the site makefile rules and whatever else, and include those files from the actual site makefiles. Then they can be edited in the wiki using existing tools. Leave the WW make rules out of the wiki. I don't know about per-wiki makefiles.
  • There may be scenarios that can break the automatic latex-compiling apparatus. I think the weakness of the current design is if A.tex includes B.tex, which includes C (a figure, tex file, bib or whatever), then C is removed from B.tex and from the working directory. Then the document can't be remade until the A.d file is removed by hand (the 'clear working directory' action on Special:ManageProject is useful here). This is because B.tex isn't a dependency of the automatic makefile A.d. It wouldn't really help to make A.d depend on B.tex because then everything would break if B.tex is removed from A.tex and from the working directory. (Note: the makefile rules have been changed since I wrote this, but I think this may still apply.)

Chroot/AppArmor/Tomoyo

  • Phasing out chroot trap apparatus in favor of using an AppArmor profile on lalashan and soon a Tomoyo Linux profile on yushan.

editing

  • the default project name when semi-automatically adding files needs to be carefully thought through and checked.
  • project link should appear when editing the page, even if it isn't previewed.
  • add two little square buttons above the edit textfield, to insert source files and project files?

internals

  • missing source file when it's a general-purpose utility file on its own page (i.e. no explicit project name, unused implicit project for its page).
  • validate project filenames in WWAction classes, and all other input as well. (I did this — make sure it's all the way done)
  • There is a DoS bug visible in my security-testing page, that causes an out-of-memory error when I edit the page. UPDATE: fixed that one, I think. Need to do some more rigorous security evaluation.
  • When $parser->recursiveTagParse() is called, it appends to the $pagetext cache and invalidates the cache, causing the whole page to go through replace_inlines() and find_project_files_on_page() another time, which is a waste of time.
  • ParserAfterStrip hook (catch_incoming_page()) not being called on ProjectDescription: page text? Why not?
  • Some of the security checking is now broken (in r48)
  • Tried to sync a source file and failed? Don't keep trying and reporting failure over and over.
  • rename functions in misc.php so their names all start with 'pf' 'wwf'
  • get all the output messages into the internationalization framework... so far only some of them are, and some are hardcoded.
  • need to check $wgMaxArticleSize before automatic edits to source-file and project-description content, and also look out for articles deleted since the last time we looked at them, the way EditPage.php does.
  • Why does make seem to succeed when it has reported 'no rule to make'
  • sort out what exactly is controlled by $peStrictFileExtensions vs. $wgStrictFileExtensions, because the code is confusing me
  • double-check makefile names are shell-escaped in make_product()

Locking

Directory locking is now in operation. We are planning to install WW on a server cluster with NFS-mounted working directories, so locking needs to be compatible with NFS.

Test directory locking on NFS. This is done (Feb 4 2010).

Interaction with LVS. Yushan's wikis are going to run on Linux Virtual Server. In this configuration, individual page requests will run on various physical hosts in the cluster, and a "service monitor" will take a given host off the available list if it tanks. We will want to make sure that if a WW job tanks while it has a working directory locked, that the lock gets removed. (this seems fine — we're using fcntl() for locks, which drops them when the process dies.)

Misc

  • When I'm on a ManageProject page, say, and then log in, I get returned to an error page because it forgets the arguments to ManageProject.
  • review for compliance with accessibility guidelines
  • path disclosure issues:
    • full paths to executables, including those in $(RESOURCES), are visible in .make.log files.
    • some are also visible in the default makefiles, which are visible to wiki users.
  • Extension to add:
    • There's one to put a 'purge' tab
    • The one for internal links with '?' parameters in them
    • Cite and its variants?

To-do list

See the feature requests tracker on sourceforge.net. (the items here should be moved to there...)

  • How to share material among all these wikis?
    • something like WikiMedia commons?
  • develop a full theory of previewing and time travel in general in WW

Restricted options for WW installs: latex-only, inline-latex-only, no-user-makefiles.

An API like MediaWiki's for import/export of project data (and other operations?).

Feature to move projects in/out of svn, git repositories.

Future version: possibly put project descriptions in a custom DB table, for efficiency, instead of in the wiki pages; and mark entire pages as source-files without using the <source-file> tags.

see LaTeXML Notes page

consider making a project-file class for use within the Project structure. Polymorphism for 'display=' behavior, parsing out included files, etc.

Automatic one-time conversion of wiki page to latex — use http://en.wikibooks.org/wiki/User:Derbeth/javaLatex maybe?

Associating parts of the HTML output back to the various source files. This would allow a sort of section-editing feature (e.g. this part of the document came from methods.tex - click here to edit that), and would allow displaying the Methods section on the methods.tex page.

Maybe an [edit] link to edit just one source-file as if it were a section? This would help with syntax highlighting.

Does SyntaxHighlighter fail gracefully when language is not installed? No. Contact its author about ways of asking the extension about supported languages.

Security: see security page

Cron script or something to clean out the old project files once in a while.

Follow the advice for MW developers on http://www.organicdesign.co.nz/MediaWiki_code_snippets, http://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker. (Maybe I've done all this by now? Haven't looked at the page in a long time)

Support for IE with MathML features

Assign DOI's to published LaTeX papers, other things?

HTML5 output

Rewrite Special:CustomUpload with MW's new upload code


Suggestions from JD

  • put unreadable files in working directory when source files are missing, helps with error reporting
  • treat image source-files the same as text files

We should consider a facility for running offline, which I claim is not the same thing as checkout.

Usability ideas

  • On our computer at least, my biggest complaint about WW is how darn long latexml takes to compile up a math-ey document. (like, one minute) Are others having this problem? Any suggested ways around this? One way around this would be to figure out a good workflow that would allow section editing, in such a way that latexml doesn't have to reprocess the whole document, just each section. Not sure what this would look like... Peter
    • This might be doable... latexml works in two stages - first it makes the LaTeX code into a .xml file and then it processes the xml into html. Depending on which stage is taking most of the time, it might be helpful to process different parts of the document into xml separately. I don't know whether latexmlpost (the second part) supports combining multiple xmls, but it's worth investigating and/or asking Bruce to add features. WuLi

Checklist for testing

(this list is outdated)

  • correct compilation and display of TeX text, equation alignment, images, figure captions + alignment, unicode characters
    • recompile paper from scratch, make sure it compiles correctly
    • make sure the alt links are there and do what they should (this should take care of testing Special:GetProjectFile).
    • change the source, see the page change
  • change R code, see image change without requiring extra reloads.
  • preview a change in source text, see the change in the preview page, don't save, go back to the unmodified page and see if it's showing the unsaved changes.
  • project name containing spaces
  • correct recreation and display of R-generated image in page, and in TeX document
  • correct display of image from Image: page in TeX
  • inline display of png image, jpg image, eps image
  • create a new page with a <latex> element or something, make sure it renders right on the initial save
  • an assortment of inputs designed to break the security
  • the various features on Special:ManageProject
  • check the security page
  • preview, edit/submit, undo, view old revision
  • Links in project-description markup.

What should be in place before ‘releasing’

(see http://sourceforge.net/tracker/?func=detail&aid=3155931&group_id=366300&atid=1527385)

Different levels of ‘release’:

make the documentation visible to the world so they can find out what the project is
  • No risk, in principle
  • Currently the documentation is mixed with a couple example projects, so this would actually be a read-only PE wiki.
  • This was done a while ago although I think we have kept it out of google. [not anymore -lw]
a publicly read-only PE wiki with example projects
  • just needs to be working pretty well ✔
  • but need to be sure PE features (and any other extensions) respect the read-only setting ✔
  • we have one of these up now: theobio/projects
Publicly editable PE wiki
  • should be generally bug-free
  • security we feel good about:
    • Tomoyo/AppArmor with a limited number of executable programs ✔
    • protection against various DoS scenarios ✔
    • validate output files for dangerous content - THIS IS NOT FINISHED
  • solid support for both MathML and plain browsers ✔
  • good directory locking and previewing ✔
  • stable policy of filename validation ✔
  • some demo pages ✔
  • not all these are necessary, though the security features are
Make the code available for download
  • Doesn't have to be a finished product. Really can release any time.
  • Prefer it not to be embarassing.
  • It's probably good enough now.
A public bug tracker for users' bug reports ✔
  • Any time.
  • Not really needed before at least one of these other scenarios.
  • Might be useful anyway, though.
Retrieved from "http://lalashan.mcmaster.ca/theobio/projects/index.php/Talk:WorkingWiki"
Personal tools
Projects
Go: