line input # irregularity

[troycheek]

Will somebody please try the following code for me and compare your results to mine:

OpenW 1 : Win_1.AutoRedraw = True
Auto a$, f$, i%, linecount%

f$ = TempDir$ + "dummy.dat"
Print "Creating 4 lines of data..."
Open f$ for Output As # 1
For i% = 1 To 4
 a$ = String$(7000 + 1000 * i%, 65)
 Print "Line "; i; ", Length "; Len(a$)
 Print # 1; a$
Next i%
Close # 1

Print
Print "Reading data with line input..."
Open f$ for Input As # 1
linecount% = 0
While Not EOF(# 1)
 Line Input # 1, a$
 Inc linecount%
 Print "Line "; linecount%; ", Length "; Len(a$)
Wend
Close # 1

Print
Print "Reading data with input..."
Open f$ for Input As # 1
linecount% = 0
While Not EOF(# 1)
 Input # 1, a$
 Inc linecount%
 Print "Line "; linecount%; ", Length "; Len(a$)
Wend
Close # 1

Print : Print "Program ends."

Do : Sleep : Until Win_1 Is Nothing
CloseW 1

This code creates a dummy file filled with 4 lines of "A" that are 8000, 9000, 10000, and 11000 characters long.  It then tries to read each line with line input #.  When I try this code, lines with 8000 and 9000 chars load normally.  The line with 10000 is truncated to 9999 and then there's an additional line of 0 chars.  The line with 11000 is truncated to 9999 and there are 1000 chars in the additional line.  It seems that line input # is reading only up to 9999 characters, then aborting on char 10000, leaving the file location pointer so that the next line input # starts reading there.  And a character turns up missing somewhere.

This code then tries the same thing with input # (no "line").  In this case, input # reads 10003 chars before aborting, leaving the rest to be read by the next input # statement, and also loses a character somewhere.

Just wanted to make sure this wasn't just my copy of GB32 or my computer.

[dragonjim]

The same thing happens on my system too.

There are two potential workarounds, both which work in pretty much the same way:

Firstly, you can use the Input$() function which is specifically designed to input strings from a file:

OpenW 1 : Win_1.AutoRedraw = True
Auto a$, f$, i%, linecount%

f$ = TempDir$ + "dummy.dat"
Print "Creating 4 lines of data..."
Open f$ for Output As # 1
For i% = 1 To 4
    a$ = String$(7000 + 1000 * i%, 65)
    Print "Line "; i; ", Length "; Len(a$)
    Print # 1; (7000 + (1000 * i%))
    Print # 1; a$
Next i%
Close # 1

Print
Print "Reading data with line input..."
Open f$ for Input As # 1
linecount% = 0
While Not EOF(# 1)
    Input # 1;i%
    a$ = Input$(i%, # 1)
    Inc linecount%
    Print "Line "; linecount%; ", Length "; Len(a$)
    Input # 1;a$ // clears the end marker added by Print #1
Wend
Close # 1

Print : Print "Program ends."

Do : Sleep : Until Win_1 Is Nothing
CloseW 1

The other is to use a similar method with Bput and Bget replacing Print# and Input$ as in the following example:

OpenW 1 : Win_1.AutoRedraw = True
Auto a$, f$, i%, linecount%

f$ = TempDir$ + "dummy.dat"
Print "Creating 4 lines of data..."
Open f$ for Output As # 1
For i% = 1 To 4
    a$ = String$(7000 + 1000 * i%, 65)
    Print "Line "; i; ", Length "; Len(a$)
    Print # 1; (7000 + (1000 * i%))
    BPut # 1, V:a$, Len(a$)
Next i%
Close # 1

Print
Print "Reading data with line input..."
Open f$ for Input As # 1
linecount% = 0
While Not EOF(# 1)
    Input # 1;i%
    a$ = Space(i%) // Make sure that a$ is the correct length
    BGet # 1, V:a$, i%
    Inc linecount%
    Print "Line "; linecount%; ", Length "; Len(a$)
Wend
Close # 1

Print : Print "Program ends."

Do : Sleep : Until Win_1 Is Nothing
CloseW 1

I'll make a note to update the help file regarding Input# and Line Input# accordingly.

[troycheek]

I'm a big fan of Bput and Bget since my Turbo-Basic XL days. If I'm going that route, I might just load the whole file into one big buffer string and cut it up using Instr to look for CR/LF pairs. Of course, it just occurs to me that for my purposes I don't really need to save the data to a file, that was just for convenience, and I could just as easily keep the data in a string array. But I thought the Input # behavior needed to be confirmed. I saw reference in the String section of the help file that GB32 when passing strings to Windows API functions uses a 1030 byte string buffer. Perhaps [Line] Input # uses something similar. (That also might explain why I was unable to pass strings as parameters when creating threads in another experiment. I was trying to pass much longer strings. Hmm. I could save the string to a temporary file and just pass the file name as parameter when creating the thread. Must check that out later.)

Along those lines, if you get bored look into creating really big strings. a$=string$(256*1024*1024,65) gives me an access violation exception error, as does any multiple of 256 MB. Attempting to create a string longer than 256 MB results in one that is "expected length MOD 256 MB" in length, if that makes any sense. However, I can create an exactly 256 MB string by using a$=string$(128*1024*1024,65):a$=a$+a$. I can't create an exactly 256 MB string by using a$=string$(256*1024*1024-1,65)+chr$(65), as that gives me the access violation exception error again. The help file says that you can create strings up to 2 billion (2^31) characters, but I can't seem to create any single string larger than 256 MB. I can create lots of 256 MB or smaller strings, but I run out of string memory somewhere just north of 1.25 GB.

Why do I need to create strings that big? An old trick from my Atari ST GFA BASIC days. Really big strings make good file buffers and other constructs requiring lots and lots of memory. I know you're supposed to use malloc for that purpose, but if the the GFA BASIC ST program ended unexpectedly (before it could mfree), the allocated memory was lost, and eventually I'd have to reboot my ST to get it back. String memory was always released back to the system when the program ended. The only down side is that you have to constantly check V:a$ every time you access the data as it's possible the string will get moved around in memory as other variables are created and/or destroyed.

[dragonjim]

When I get a minute or two, I'll look into the matter of strings but 'shooting from the hip' I think you'll find that, while a string can technically be 2 billion characters long, the functions that GFA uses to handle them (like Input#) may not be able to deal with them. I know that TextBoxes and RichEdit boxes can take roughly 1 billion characters before invoking an error, and as they are BSTR (wide character Unicode) that would fit with the 2 billion limit.

Edit Did a quick test and I can get up to 2^28 using the routine below:

Local a$, m As Int32, n As Int32
a = "A"
For n = 0 To 27
    a = a & a
    Debug Len(a), Log2(Len(a))
Next n
m = Len(a)
Do
    a = a & "A" : Inc m : If Len(a) < m Then Exit Do
Loop
Debug "Max: ";m - 1;" Len(a) =";Len(a)

[troycheek]

2^28 = 256 * 1024 * 1024 = 256 MB, right? I never used TextBox or RichEdit, just really big strings.

[dragonjim]

256MB it is (another edit for the help file); Len() works directly from the string descriptor so I'm reasonably sure that's the limit.

For a fixed string the limit seems to be (2^20)-1 or just one short of 1MB as stated in the help file.

If you are manipulating the data in-house (rather than passing it en bloc to a .dll), you could load it into a string array. Otherwise it is good old Malloc which you'll be glad to know works much better under Windows than on the Atari ST(E/FM) where it was one of only a very few annoying bugs. For some reason, GFA never ported the instructions like On Break (I am unaware of a way in GB32 to intercept the Ctrl-Break combination) and Every or After (fortunately, there's a workaround for them) which I used to use regularly from good old GFA 3.5 for ATARI. But then again, I suppose you couldn't have 256MB strings on the Atari either.

[troycheek]

Well, I was going to complain here about mAlloc(-1) not working anymore and just having to guess about how much memory is free to allocate, but it looks like someone has rewritten the help file with a workaround since the last time I checked. Of course, although I have 8.0 GB of Available Physical Memory, I can still only allocate about 1.25 GB max with mAlloc (varies between runs). That's roughly as much as I can get with a string array, so I'm betting that's some kind of limit built into GB32. It being a 32 bit application, it's limited to 2 GB total anyway, I think.

My Atari ST was limited to 4 MB and my Falcon030 was limited to 14 MB total RAM, so the super strings were obviously smaller than that. I was mostly dealing with files that would fit on a 720 KB disk, so creating really big strings wasn't a problem. I was trying to write a GB32 program to calculate CRC32 on some files on my hard drive when I came across a virtual disk that weighed in at over 10 GB, which led me to really big strings, mAlloc, and a lot of experimenting about limits. Never did get around to finishing that program, now that I think about it.

Damn. I just realized that this computer has more RAM than my first Atari ST hard disk drive had total storage (16 GB vs 30 MB). The Falcon030 came with a built-in 80 MB drive.

[dragonjim]

I remember the Atari STE well, although I never got around to progressing to the Falcon. It was much-maligned as being too little too late in the battle against the Amiga and ultimately the PC with MSDOS and Windows 3.11 - was it really that bad? The specs looked quite impressive, although I seem to remember that the criticism was due to their rigidity and the lack of customisation options. Thinking back, they were simpler days back then: one program running at a time until the arrival of MultiTOS and limited or no internet.

As to accessing more than the 4GB Win32 limit: have a look at this page; I tested the download some time back and it seemed to work fine with no nasty adware complications - that said, as with all suggested links, use at your own risk.

[troycheek]

The Falcon030 would have been more impressive had it come out a year or two sooner, and unfortunately Atari dropped out of the home computer market soon after to focus on game systems like the Lynx and Jaguar. There was nothing wrong with the machine itself, though there were few programs and games that took full advantage of its capabilities, everything being written to be compatible with the old ST/STE machines. I think it ran all the TT030 programs, but there were precious few of those as well. And of course there was poor little me trying to get GFA BASIC 3.6TT to work in Falcon030 graphics modes.

I've experimented with Win32 and the 4 GB limit. One of my old machines still takes 5 extra seconds to boot up because it pauses to let you select between standard Vista and a Vista which is patched to use more memory. Unfortunately, the drivers for some of the hardware freak out when presented with 4 GB or more of usable RAM. However, the problem of which I speak is not the total amount of RAM seen by the OS, but the amount of RAM seen by each individual program even when running on a 64 bit OS. GB32 programs seem to be limited to about 1.25 GB whether by mAlloc, super strings, or huge integer arrays even though I have 8 GB of RAM free. Threads don't help. I can spawn extra processes, each with their own 1.25 GB limit, and communicate with window messages, but that gets really complicated really fast.

Back to the original topic, where I couldn't use line input # to load arbitrarily long lines of text into a string array, I apparently overlooked the Store and Recall commands which are designed for exactly this situation. Must do more testing.

[dragonjim]

I think you are right about the Falcon: it did come on the market about two years too late. That said, no rigid spec computer survived the early-90's trickle to PCs which became a stampede on the arrival of Win95. And as to what you say about games being limited by having to run of STE's, I have some sympathy. As an STE 1040 user with a super-dooper blitter chip, it was incredibly frustrating to have to put up with games which were developed for the blitter-less STFM (and STF and STM if there were any of those still around), especially when the STE had the ability to not just equal the Amiga but in certain areas beat it hands down.

Anyway, back to thread...

Store and Recall are good options but they have a similar problem to (Line) Input# - there is a limit on recall of 9999 bytes per string element; one byte is then lost and the remaining bytes get moved into the next element and all the other elements are shunted up one.

This is best shown by the following example:

Local a$(100), n As Int32
For n = 1 To 100 : a$(n) = String(12000, "A") : Next n
Open App.Path & "\store.tmp" for Output As # 1
Store # 1, a$()
Close # 1
For n = 1 To 100 : a$(n) = "" : Next n
Open App.Path & "\store.tmp" for Input As # 1
Recall # 1, a$(), -1, n
Close # 1
Print Len(a$(1))
Print Len(a$(2))
Kill App.Path & "\store.tmp"

If you can wait a day or so for the next build of the help file, there is a workaround to this which someone came up with some time back before I started doing the help file. I have managed to track it down and I will have to test it and write it up (it'll be on the Store/Recall page under Known Issues)

[troycheek]

I'd suspected the 9999 byte limit would also apply to Recall, but hoped I was wrong as I'd never used it. I often find GFA commands and functions that do exactly what I need to do, but only after I've worked out how to do it with other commands. (I came up with three different ways to save a screen image before I discovered the SavePicture command, for example.) I swear I've gone through the help file from start to finish multiple times, but I still find new commands every time I go looking.

The Recall workaround included in the help file Build 7 looks solid and suspiciously like some of the things I've programmed in the past. Kudos to you and Roger Cabo.