# Where should binary files be stored?

Which poem is this? was recently asked, and the poster's friend gave him a binary file to help solve the puzzle. StackExchange already has great support for images, but where should people put other resources needed to ask (or answer) a question? A place like Pastebin for files would be great that allows anonymous uploads up to some size limit since most people probably do not want to base 85 encode or decode their data. Alternatively, an application could be developed so people could transmit their data through steganographic images.

What do you use to transmit non-pictures to people answering or asking questions?

There are many good points raised in niemiro's answer, but I wanted to emphasise a couple things.

• Puzzles should be self-contained - this avoids any of the considerations raised regarding the possibility that data goes missing if its hosting location goes off-line. We strongly recommend that, if it is possible for all the information to be provided within the posting of the puzzle itself, that it be put there. In particular, for the puzzle referenced in the question here, the data in question didn't have to be encoded the way it was; rather than requiring someone to know that "lzma" and "pickle" are meaningful to enable them to recover data from an encoded binary file, the data could have been provided in a different way (perhaps even not encoded at all) that would have allowed it to be more readily presented and used in text form within the puzzle posting proper.
• Images (per se) are fine; they already have a home at i.stack.imgur, and the SE interface provides integration to streamline use of that hosting site. They are displayed directly in the browser and, even when used to steganographically embed data, extracting RGB- or bitwise-encoded information can be done with online or local tools without particular risk.
• Other data is more troublesome. We never want users to download data of unknown provenance to their own computers, as that's a Really Bad Idea™.
• If there is data required to solve the puzzle, but which is difficult or impossible to present in the puzzle body or in an image file, it is very likely that this puzzle is not going to be appropriate here, as required data is not self-contained in the posting or its images.
• If there is optional data which can provide solvers useful templates, data visualization, or the like, this data can be provided but should be given in a way that does not require downloading binary files make use of it. For example ...
Type of data How to host
Bulk text pastebin can display bulk text in user's browser, and can be cut&pasted safely to local machine if desired
Program code online runtime environments, like tio.org, allow code to be run and even modified, and output viewed, without downloading code or installing an interpreter
3D models Online viewers, e.g. http://www.viewstl.com

These allow data to be viewed/manipulated in a browser without downloading it, and generally allow the text of the data to be cut & pasted locally if desired, all without having to download a binary file. The sites specifically named here are considered reputable and don't require users to sign up to view content hosted there.

Keep in mind these should really only be used for optional data; there is no guarantee that any of these off-site resources will remain available, and we don't want a puzzle to be lost because vital data disappeared.

• Executables are never okay.
• Archive file formats are not okay. That means .zip .tar .tgz .7z .rar and the like. These not only are binary formats themselves, they can contain other binary files as well, and as they contain entire (named) files there is a potential for existing files to be overwritten, for harmful files to be included amongst innocuous ones, and/or for files to be saved in unexpected and unwanted locations that end up causing harm later.
• Encoding executables or archive files in other ways (encoded as text, smuggled in image steganography, etc.) is still not okay (and attempting to hide potentially harmful content this way is a great technique for getting unwanted attention from a moderator!)
• Some people engage in puzzles that originate off of StackExchange but then end up bringing them here. Some of those puzzles involve binary data outside of the asker's control. If and when they discover such a requirement or help for solving a puzzle, what are they supposed to do: (1) base 85 encode the data and display it in the question, (2) host the file on some undetermined external website, (3) delete the question in the case it is unsolvable without the binary data -- impossible if there is already at least one answer, or (4) do something else -- good ideas are welcome for submission. – Noctis Skytower Sep 4 '17 at 13:14
• If (1) is possible, then (as this answer says) it should be included in the posting so the puzzle is self-contained. The gist of this answer is that (2) should not be done, and (3) should be, if externally hosted data is required to solve the puzzle — and if answers preclude the community from using moderation tools to delete it directly, then they should flag it for Moderator attention, as the Mods are able to delete it anyway. There will be puzzles off of StackExchange that will not be appropriate here; we're perfectly fine with that being the case. – Rubio Sep 5 '17 at 0:16
• So, do you believe that imgur mazes should be prohibited? – Peregrine Rook Sep 5 '17 at 3:49
• @PeregrineRook Good question. I don't think they should be. Because of the integration of i.stack.imgur hosting with SE, we treat images hosted there as "on-site". So while an imgur maze does not place the entirety of the puzzle literally in the text presented in the original posting, when combined with images which are (effectively) hosted here, it is sufficiently self-contained — in that it does not rely on "off-site" resources that could go offline — that they should be ok. (That doesn't mean I don't think imgur mazes have other issues — they do — but this isn't one of them.) – Rubio Sep 5 '17 at 3:56

In my opinion the primary considerations are:

1. Longevity of the data. It's really important that critical information integral to a particular question's solvability does not go missing after six months or even after six years.

2. Accessibility. Nobody should be required to register an account at an external website in order to download the data.

4. Verifiability. The binary data file should be, where applicable, easy to reconstruct and easy to verify that the reconstruction has been successful.

Recommendations:

1. All binary data should be, wherever remotely possible, embedded within the original Puzzling Stack Exchange question (see advice below on how to do this).

2. As an alternative for larger files (presently for snips up to 0.5MB), pastebin.com is generally seen as a reputable and safe website for storing plain text data.

3. For even larger binary files, any reputable file sharing website which satisfies the above above primary considerations should be considered acceptable. My personal recommendation would be to share a link to the file hosted on Microsoft OneDrive, Google Drive or a similar service. I would advise users to consider registering an account specifically for their Puzzling.StackExchange activities in order to separate question content from their personal data to help ensure continued longevity of access.

Unacceptable:

1. I would advise users against using their own personal websites to host data (e.g. niemiro.co.uk for me) because these websites are liable to go offline within a couple of years, are often on slow hosting and may not be able to handle the traffic Stack Exchange could redirect towards them.

Storing binary data in plain text (for within the question body or on pastebin.com):

1. Prefer a hex dump. Open the binary file in any reputable hex editor and post us a complete hex dump of the file contents.

2. Include a SHA256 hash of the original file within your original Stack Exchange post.

3. Ensure that you are personally able to reconstruct the binary file with identical file hash to the original prior to posting the question.

• Would you like a tool developed to make your three-step process easy to do? That would allow incremental improvements to the process and generate feedback. – Noctis Skytower Sep 1 '17 at 13:17