User avatar
iljitsch

Posted Wed Feb 02, 2022 8:22 am

If the following sounds like heresy, my apologies in advance.

But what if at some point in the future, you want to open a file that you created on your Amiga in its heyday... and your boring modern computer says "huh?"

To avoid that situation, I decided to go over all my old Amiga data from the 1990s (and even early 2000s it turns out even though I got my first PC at the turn of the millennium). Here are some experiences that may benefit others.

The idea was to convert to formats that are widely supported today, and if at all possible, convert to something that allows further conversion to something else without complexity or loss of quality. So for images, PNG is better than JPG. Both are widely supported, but JPG is lossy and PNG is lossless.

Identifying files

On a Mac or other Unix-like system, the file tool is really useful for identifying file types, especially for ones that don't have an extension and will thus not be recognized automatically by modern OSes. For file types that didn't need conversion I generally made a copy with the appropriate extension, in all cases leaving the originals alone.

Images

Most Amiga images are in IFF ILBM, and most of those can be converted to PNG without losing any data. I think the main issue is color cycling. Perhaps revisit IFF images with color cycling and convert them to GIF or something else that animates.

I used PNGTool for this, which will take any image for which you have a datatype. I did end up with some broken images, I have to further check if there is something wrong with the originals or if PNGTool didn't work properly.

Faxes

I used my Amiga to send and receive faxes for many years, so I had a bunch of those to covert. With the right FAXX datatype off of Aminet that should have worked, but in practice PNGTool wasn't able to convert many of them, and opening them in Multiview was problematic due to the large size and only 2 MB chip RAM. But fortunately I still had GPFax installed from way back in 1993 and it opened and printed all the faxes just fine. Later I switched to TrapFax and that also still worked just fine. Not sure if I'd be able to install either of them from scratch today, though.

Samples

Most Amiga audio sample files are in IFF 8SVX. I used Sox to convert those to WAV ( -t wav ). Sox (also available for the Amiga) recognizes the 8SVX files and the sampling rate automatically, so that makes conversion nice and easy.

I got an ISDN card for my 3000 in 1995, and for four years that card served as my answering machine. (ISDN = digital phoneline a few years before we had broadband.) My messages were recorded in native "A-law" format (would be "mu-law" in the US) and that's a raw format, so I had to tell Sox " -t la " (not " -t al ") to make that work. Pretty crazy but fun to listen back to messages from 25 years ago!

Office-type documents and printing

The nice thing about the file types above is that they all convert to something that does pretty much the same thing today. For office-type apps such as word processing and spreadsheets that's much more difficult. Basically you can go through one or more conversion steps to get those document to load in modern counterparts, but you lose a lot of formatting that way, but you can still edit the documents. Or "print" the documents and retain most of the formatting, but not the ability to edit.

Unless it was very clear that one solution was the appropriate one, I generally just did both.

I initially tried to "print" to a PostScript file and then tried to convert that to PDF on my Mac, but that pretty much never worked. A better solution was printing using a Laserjet printer driver and capture the output using the CMD tool, and then use the rather obscure PCL to PDF tool convert those files to PDF for posterity. Resolution isn't great and no color, but pretty decent and not unduly large PDFs. Perhaps color will work with a more advanced PCL printer driver but I didn't bother looking for those.

Wordworth and ProCalc

These would be the office type applications mentioned above. I have Wordworth 6 on my Amiga in good working order. Main issue was the extremely slow printing to PCL. I also converted lots of files to RTF, which is technically a proprietary Microsoft format but in practice lots of stuff can read it, and you at least get the very basic formatting across.

Wordworth also does WordPerfect files, so I was able to use it to convert a bunch of those that I still had, mostly from college.

A big problem was all the different fonts. Files made to be printed on one printer would switch to substitute fonts when changing printer drivers, and usually not ones even closely related. And although RTF conserves font names, in practice fonts were replaced by something pretty random there, too. I guess you can go into the RTF file and manually change font names...

With ProCalc, no easy RTF option, but only Lotus 1-2-3, which was already pretty old back in the day, and I don't think anything I have on the Mac or Windows can read those, but with some effort that should be doable. Interestingly, ProCalc prints in text mode in portrait, so that's nice and fast and mostly just text that you can recover. Or in graphics mode in landscape, which is much slower and you get to see the Amiga's bitmap fonts in supersize pixelated glory. :(

Text files

Of course we've had plain text files since the dawn of computing, and I bet we'll have them to the dusk of computing, too. If you're on a Mac or Linux or something like that, plain ASCII files made on the Amiga will work just fine. (That .txt extension does make things easier, though.)

On Windows, you may want to convert the line endings.

But mostly, you'll want to check if those files have any "high ASCII" characters in them. For instance, in your book reports on the Brontë sisters. The Amiga used the ISO-8859-1 encoding for those characters, while we now use Unicode, mostly in the form of UTF-8. So I made a little script on my Mac that uses the iconv tool to create a copy with UTF-8 encoding and the .txt extension.

But check with file first, if it says that a file is ASCII, it doesn't have any offending characters and no conversion is necessary.

User avatar
intric8
Seattle, WA, USA

Posted Wed Feb 02, 2022 9:48 am

This was an absolute pleasure to read. Thanks for posting.

I can't believe your old Amiga was used as an answering machine! Were the audio files (messages) saved to tape? How did that work exactly? If they were digitized, they must have been really heavily compressed! Did it require a lot of hard drive space? Really cool.

(And what was your machine's message to callers? Was it your voice? The computer's?)

User avatar
iljitsch

Posted Thu Feb 03, 2022 5:09 am

I had the BSC ISDN Master card. This hooks up to the little ISDN terminator box using (pretty much) an Ethernet cable. It also has an audio in connector and a connector for a telephone handset. Just the handset.

The phone network started becoming digital in the 1960s. This worked by sampling the audio at 8 kHz with 16 (14?) bits compressed down into 8 bits. So that's 8000 x 8 = 64000 bits per second. With ISDN, you as the subscriber also got a digital connection (over a normal copper line) with two of those 64000 bps channels that you could use for data and/or phone calls. The data part works using the "bscisdn.device" as a replacement for serial.device. Works the same, just faster. 8-)

The phone part was handled by a separate program, WilhelmTEL:
WilhelmTEL.gif
This has a phone book and lets you place and receive calls using the headset connected to the back of the computer.

The card handles the digital/analog conversion between the handset and the ISDN line. And you could also record the conversation or play an audio file through the call.

When you didn't answer the call, the program would of course record messages, which would be stored as files on the computer. I told a colleague about my new toy and he immediately called my number and left this message (small WAV file), which doesn't make any sense in Dutch either.

You could record your own messages, also to 8 kHz a-law / mu-law files. This was mine. "Hello, this is Iljitsch van Beijnum. I'm not here right now, so tell it to the computer and I'll call you back."

User avatar
3D-vice
Germany

Posted Thu Feb 03, 2022 10:29 am

Great write-up! Thanks for sharing. :boing: <3

User avatar
Gernot66

Posted Wed Feb 09, 2022 9:32 pm

Will you mind if i post something to "image conversion" and "SFX conversion"?
OK you can't i posted it already ;)

PNG is for sure the right choice and well there is not many image on my windoze which isn't in this format (check this and you will know why https://photos.app.goo.gl/ReS6SnCsd4Vw5Qw991) and no i HATE (a thick fat HATE) DDS textures they are more lossy as jpeg!
(an 8 bit image will be reduced by DDS to about 128 colors, that's the "compression" they talk about a complete loss of half of the data just to make it possible to have a useless high graded alpha channel in the same bitmap. I experienced for alpha blending that 32 colors are more as enough, sure PNG can't store or calculate reduced mipmap images but that a small advantage compared to the loss of data and the alpha is either 1 or 8 bit, neither that i have a performance drop by using PNG instead of the by my mates chosen DDS on my miserable notebook resumeé: DDS has no advantages for me resp. it has no advantage for OpenGL driven 3D).

NEW DOESN'T EQUAL ALWAYS TO BETTER (you know it)

To convert IFF to PNG and back i use XnView, it never failed (for the ~300 images which i converted), my common image editor/paint program is as you can guess GIMP, it had an IFF exporter for 2.08 but it fails in 2.10 and apart from that it never was as good as XnView, GIMP exported them always as 8bit images but since i run even a OS1.3 recently most images have 2 bitplanes to be used with SimGen (or WBPic). XnView keeps the 2 bitplanes but has no idea about 4, however 4 and 8 color (next recognized for XnView is 32 colors) will have a stripped palette and PNG can very well respect this respectively it is pretty the same here (unlike GIF which is always 8bit). In other terms an IFF using only 2 or 4 colors will also have only set these two or four for the exported PNG and vice versa, this helps to keep the palette DPaint (or similar) won't have to recalculate the palette for fewer colors.

Portable Network Graphics how aged is this format?
Does that play any role?
They won't create a better tomorrow and not in hundred years, it stores a bitmap and it stores it lossless compressed.
And to be honest i neither can see that IFF/ILBM would become obsolete, i mean they store bitmaps that's all and any new format can't do it better as to use a huffman compression. They only reason to have new formats is to make the old obsolete, there is no real gain possible!
(except you cheat like for DDS and claim like nowadays every nine to fiver does how fantastic and unique and million times...
...worse it is).
Well just as long until someone examines the result with a looking glass.


To the SFX wave and all that surf,
In general one could say about wave formats the same as about images "it's a bitmap and to store this bitmap not many possibilities you have except to store it byte by byte".
Thus in advance any "new" format can't be better as what we have since it can't be made different as to store them byte by byte, except compression but also here it is the same huffman code which is used from A to Z.
Thus i don't htink that your 8SVX will be ever obsolete because they do what they was meant for.
And well what have we if we peep into them?
The same data for all, differences are mostly the use of signed samples for the Miggy while doze prefer unsigned and 16 audio has to be either "little endian" or big "endian" (or as i would say read bytes backwards (little endian) and read bytes forward (big endian)).
8SVX would be a pretty good format IFF EA's interchange file format, likewise SB for VOC they thought about a lot of use for games even if the features was rarely used. Intentionally, as i have read, 8SVX is to store instrument samples, but apart from this the format (header structing of IFF files) offers things (or would) like to set loop markers and loop times, thus a SFX could have a start and loop in itself in a smaller section, it would allow to store the volume, channel use, a lot of things and it is extendable with everything compliant to IFF. What it can't what in example VOC could is to suppress silent passages to zero and to store a silent passage as instruction rather as data, but well apart from the original SB16 software none can make use of the many features of VOC. Similar for 8SVX i guess its capabilities has never been fully used thus obsolete if something isn't explored right yet?
Stereo, VOC as example wasn't and well you know 8SVX neither, but since it is a format to store instrument samples there is an addressing for a so called "high octave" of the sample this had been abused for the a bit strange 8SVX stereo.

The idea is right - the execution bad.

To convert from and to 8SVX i use no other as i use else - Audacity.
Audacity (respectively FFMPEG) will resolve the special stereo of 8SVX proper (with which the native machine has problems as well, later more). Files can fail to be resolved proper as stereo but one can claim what fails to be resolved here fails even on the miggy (certain sound players are far more picky as audacity about "stereo"). Audacity can't export to stereo samples, one could claim "because stereo doesn't exists for 8SVX" which is neither false nor true.
In fact you will find NO program who can do this and have to use a tool from the miggy i.e. "join2stereo".
It's no magic "join2stereo" simply takes the two mono samples and glues them together while writing to the "high octave" address the address where the let's say "other", because it could be theoretically be any of the four, channel starts, you can do that as well with a hex editor, it's no magic...
...and here is already the source for the errors.
That is a) the stiff limit that stereo samples need to be even in length and there is no exclusion to this
even more worse
the header size as to be uneven in length else the whole file will be uneven in length and then a possible Stereo won't be interpreted right and one channel is played back after the other like they are stored, Audacity and any player on your miggy will fail to play them proper back, but as i look at this problem it could be solved right it just was solved sloppy.
A header is just a header and header A is to me like header B, in general samplerate, bitdepth, little or big endian (for some formats) stereo flag, length in bytes.

What i guess the dudes made wrong;
In the 8SVX header you declare the header size by "total filesize from here", as usual the entry code to declare what it is is disregarded it won't be count to filesize, no matter which format in this all behave the same, after this follows the string, that's why i need to declare a size "8SVX", then to shortcut it samplerate and all that tidbits, After you would be quite free what you declare in the header, it's IFF! The first keyword entry after "all the tidbits" has a leading "FF" as length and i assume - i highly assume this sloppy use is part of the reason for the failure, it declares simply a "unspecified length" any of the following to read out data has a proper length before the data follows. Finally after all that (it could be nothing since AUTHOR,NAME,COPYRIGHT etc. all this is not of need just to play back) the keyword DATA and the length from here start the audio stream.
The main problem is now that if the header has a total length which has an even count the total file size will have an odd number and this provokes the error.

Personally i would stick the files together and instead to write each time a header which uses the filename as NAME to use no NAME since the name is what changes from even to odd, or to use a fixed space for the naming i mean 31 bytes would be fairly enough for a name. Same for AUTHOR maybe to keep the ones name who put his dirty fingers on the data, 31 glyphs are fairly enough, besides the exporter of Audacity uses exactly such a template header with always the same size, no NAME in it therefore as AUTHOR a fixed string of the developers name, IFF is flexible enough to accept if there is no filename stored in the header.
However the stereo can still fail even if you respect the odd length of the header, it seems ideal is likewise it would be 16bit to up or downsize by always 4 bytes (i assume since a name with three chars works, five not, seven yes but nine not, at first glance you get the impression it has no rule but the rules is "4 bytes" e.g. 3,7,11 and so on, if i look now at what i wrote it's obvious since we have for stereo two sequential joined tracks thus the resulting "gap" is 4 bytes (two times two bytes) and would be for 16 bit data 8bytes.
This is badly done, "join2stereo" (btw i haven't found any other good to use from cli) will argue if the input is odd but then it totally disregards the above rule and any exported stereo 8SVX with a filename length of 4,5,6,8,9 etc. bytes will fail to be played back proper as stereo, one player so far can correct this common error "SOUND" (gramma software, RL Stockton) he implemented a stereo switch which is (undocumented by him what it is for) exactly to correct this error, while it will then disregard the header length and play it together with the data which results in a short noise at start.

"Meiner bescheidenen Meinung nach" ;)
In my humble opinion it shouldn't be such a big thing to write something which works always and if i have to script it (goddammit!), it's neither magic to poke a few variables in the header.
Erm alright i leak a little of a few proggies such as a lightweight database handler it just has to read/write data and respect undefined field separators, a header is like a short database - any, no super fancy "i can do all" just something which comes close to "nset" which i used countless times in my doze (in fact i hacked nset, it had as fixed field seperator with highest priority "space" (0x20) but dammit "space" can be in any string and since i don't like to manipulate a stupid brokers table it's useless, i changed it to "semicolon" as fixed field seperator with highest priority any other you can define but one is given (there is a reason for this but i would have to lookup what i experienced then).
A header is easy to read out and to write back to it, and it should be no problem to make a "four byte rule", in other terms to fill a string if it doesn't matches the above stated rule of 3,7,11...

If that all fails i write a new header type :)
you see audio streams aren't as complicated as images to be honest and in fact i wrote one - a scripted header and it works fantastic (in DOS).

A last word to this (audio) mess,
You can use a web service to convert from and to 8SVX, personally i wonder what they perform with the feed files because they show exactly the same error as when i convert it using "join2stereo", rooted in the same the length of the name string.

Yeah this makes it obsolete, indeed or as one replied my in the audacity forum "which developer should have interest to write an export plugin for this OBSCURE format"

cough (three times)
It is not as much or less obscure as any else PCM wave format, it's pretty the same as any.
In what do they differ?
In general only in the header the data is always a raw PCM sample.
If 8SVX is obsolete then the whole PCM wave construct is obsolete and i guess that is wrong.

But "IFF"
it's not
;)

User avatar
iljitsch

Posted Thu Feb 10, 2022 12:46 am

Wow, that was an epic rant!

Do you have an example of a stereo 8SVX file? I want to see how Sox handles it.

If you look up stuff like AIFF, WAV and PNG, you'll often see words like "inspired by IFF" or "hunk structure like IFF". It does seem that "not invented here" syndrome got the better of a few companies in the 1990s.

But I do think it was right to replace IFF ILBM, as the structure where you have separate bitplanes quickly becomes a performance bottleneck. Especially with 8 bits or a multiple of 8 bits per pixel it's just no good.

User avatar
JohnNovak
Brisbane, Australia

Posted Sat May 21, 2022 5:26 pm

Probably it won't interest the non-coders among you much, but I wrote a little library that can handle RIFF files (the Microsoft/Apple extension of the format with little-endian byte order support). It's a lesser known fact that WAV, AIFF, AVI and MID files are RIFF files (there are lots of others), which is basically 99% IFF. So the legacy of IFF as a general container format definitely lives on. Google's WebP is another good recent example.

https://github.com/johnnovak/nim-riff

Originally I just wanted to write a WAV reader/writer, which led me to write a RIFF writer. In fact, I really like IFF/RIFF; it's a nice simple format to store tree-like hierarchical binary data, so I'm using RIFF in some of my own projects as the data format.

User avatar
JohnNovak
Brisbane, Australia

Posted Sat May 21, 2022 5:36 pm

iljitsch wrote:
Thu Feb 10, 2022 12:46 am
If you look up stuff like AIFF, WAV and PNG, you'll often see words like "inspired by IFF" or "hunk structure like IFF". It does seem that "not invented here" syndrome got the better of a few companies in the 1990s.
It's a bit more than that, from Wikipedia:
Resource Interchange File Format is a format developed by Microsoft and IBM in 1991 that is based on IFF, except the byte order has been changed to little-endian to match the x86 processor architecture. Apple's AIFF is a big-endian audio file format developed from IFF. The TIFF image file format is unrelated.
That was a necessary change for x86 CPUs as IFF really assumed Motorola, so there were many little details hardcoded into the format to make processing of IFF files as fast as possible on Motorola -- and Motorola only. As I see it, they respected the original IFF standard and made changes only where it made sense, and they've even kept IFF in the name, plus it's an open format, so I see no problems.

Coming up with an entirely different format to serve the same purpose instead of just extending IFF, now that would've been the "not invented here" syndrome, not this!

You mentioned PNG, they extended the (R)IFF standard even more, e.g. indicating in chunk IDs whether it's an optional or mandatory chunk (whether it starts with a lowercase or uppercase letter, respectively), they've also added checksums to chunks, etc. So again, they definitely did not want to invent a new format, but extended (R)IFF in a sensible way for their purposes, so calling it "IFF inspired" is very much correct (and I think they did the right thing by not inventing something entirely different just for the fun of it).

Another shortcoming of both IFF and RIFF formats is the inability to store data larger than 2GB in size. (Technically, RIFF allows 4GB chunks, but virtually all software out there is broken and assume signed 32-bit integers, hence the 2GB limitation... sounds similar to the FastFileSystem bug/limitation, right?). Wasn't a problem in the 90s, it's very much a problem now, at least in professional settings (it lead to the invention to Sony's Wave64 (W64) format, for example).

User avatar
BloodyCactus
Lexington VA

Posted Sat May 21, 2022 8:38 pm

the big problem with IFF + RIFF, they have a large ease of "readability" that allowed anyone to abuse it by adding their own chunks.. RIFF REALLY suffered, people added all kinds of crap to WAV files. some people understood about head lengths in the chunks, some did not. so you get chunks with and without the correct size added in or not. chunk names, or lack of etc. such a nightmare trying to have robust WAV file code..

its a shame there wasnt a bit more governance or oversight over it but then it wouldn't be so extensible.

I have a love/hate with iff + riff :)

User avatar
iljitsch

Posted Fri May 27, 2022 7:37 am

Was the byte order thing really such an issue that it warranted incompatibility? I believe some other formats solved that by allowing either one on write, and then on the same system reads will be fast and on a different one there's a bit of extra overhead.

(BTW I recently came to the reluctant conclusion that little endian is the superior byte order. This was after looking into the 68000 CPU architecture for a blog post. The 68000 has eight general purpose registers, and those can be used as 32-, 16- or 8-bit registers. With Motorola's big endian architecture that means that the first byte is different in each case. With little endian, the first byte would always get the bottom 8 bits, so that would be more logical.)

I don't think 2 or 4 GB was an issue for IFF on the Amiga as the whole filesystem can only support 4 GB anyway. :mrgreen:

As for a plethora of strange chunk types: as long as they are implemented correctly (with a correct length being an important part of that!) you can just ignore them, that's the beauty of the system.





Return to “Software”