INFO: Sjouke's Blog: File I/O using Get and Put

[(X)]

gfabasic32.blogspot.com/2023/08/file-io-using-get-and-put.html

[Roger Cabo]

Thanks to Sjouke for his kind work on this:


I encounter challenges when recommending the behavior of saving a multitude of individual data or variables using GET/PUT or Out Type n, a [,b,c...]. Without a mechanism to identify the position of each variable inside the file, managing small sets of data, such as up to 10 or 20 variables or large data blocks, might be straightforward and efficient.

However, the complexity grows exponentially when dealing with vast amounts of data, such as variables, string arrays, and types, along with hundreds of single variables for each form containing data handles. Such a scenario can lead to significant difficulties in management and retrieval.

Consider the following example: Suppose you have stored approximately 100 different variables, and you wish to add a new variable related to the first one stored at the beginning of the file. If you append the new variable to the end of the existing package, retrieving the data of this newly added variable requires reading through all the previously stored content every time. This process becomes increasingly inefficient as more data is added.

If I'm wrong in this case correct me.

Recognizing these challenges, I developed an IO HASH Soup solution some time ago, which is available here in the forum. This approach aims to streamline the storage and retrieval process, enhancing the overall efficiency of managing large datasets.

SoupTool_V2.G32 (50.83 KB)

The program is a specialized data management tool, designed for loading and saving variables, including Int64+32, Double, and Strings. It supports both single variables and multidimensional arrays, with the dimensioning of arrays being automatically recognized. The main features and functions are:

1. Loading and Saving Variables

• Supported Data Types: The program can load and save Int64, Double, and Strings.
• Multidimensional Arrays: It supports the storage of up to five-dimensional arrays, with the dimensions being automatically recognized.

2. Efficiency in Retrieving Data

• Direct Access: By utilizing a hash field, the program can directly access the required variable without the need to sift through unrelated data. This drastically reduces the time needed to find specific information.
  
• No Sequential Search: Traditional methods might require scanning through files in sequence, which can be time-consuming. The use of the hash field eliminates this need, offering a more efficient way to retrieve data.

3. Identification by Name

• Naming Convention: Variables are stored and identified using their names, making it intuitive to find the data you need.
  
• User-Friendly Approach: This approach mirrors how we typically organize and retrieve information in everyday life, using names and labels, making it more accessible even to those not deeply versed in programming or data management.
• Significant advantage: Of the program is the ability to specify variable names directly when saving. This feature not only simplifies the process of identifying and retrieving data but also adds a layer of organization and clarity to data management. For instance, an array can be named as eg. String Array Type Name "MyStringArray$(1,2,3,4,5)", making it immediately clear what the data represents and how it's structured. This name-based approach enhances the user experience by providing a more intuitive way to interact with the data. It allows for a seamless connection between the code and the stored information, fostering a more efficient and user-friendly environment for managing various data types and structures.

The ability to specify variable names directly when saving. This feature not only simplifies the process of identifying and retrieving data but also adds a layer of organization and clarity to data management.
For example, consider a five-dimensional INT array defined and saved as follows:


Global abcdef$(0,1,2,3,4)
SoupSaveINT_ARR("abcdef%", abcdef%())
Here, the array is saved with the name "abcdef%(1,2,3,4,5)," making it easy to identify what the data represents.
Later, when you need to access this data, you don't have to remember complex indices or sift through unrelated information. You can simply load the array by referring to its name, as shown in the code below:


SoupLoadINT_ARR("abcdef%()", abcdef%())
This name-based approach enhances the user experience by providing a more intuitive way to interact with the data. It allows for a seamless connection between the code and the stored information, fostering a more efficient and user-friendly environment for managing various data types and structures.

4. Flexibility and Adaptability

• Support for Various Data Types: The direct access method supports various data types, including Int64, Double, and Strings, providing flexibility in handling different kinds of information.
• Automatic Recognition of Dimensions: For multidimensional arrays, the program automatically recognizes the dimensions, further simplifying the data retrieval process.
  
  

5. A bit of Security

• Encryption of Keys: An additional level of security is provided through the encryption of the keys.

6. Direct Access Through Names and Hash Field

• Efficiency in Retrieving Data: By utilizing a hash field, the program can directly access the required variable without needing to sift through unrelated data, drastically reducing the time needed to find specific information.
• No Sequential Search: The use of the hash field eliminates the need for scanning through files in sequence, offering a more efficient way to retrieve data.
  
  

Summary:

The program serves as an effective tool for loading and saving variables in various formats, including Int64, Double, and Strings. The ability to handle multidimensional arrays, with automatic recognition of dimensions, makes it a flexible and powerful solution for data management. The identification by name, type and structure safety, the encryption of the keys, and the direct access through names and hash fields contribute to the user-friendliness and robustness of the tool.

Functions:



   • SoupInit   ' Initialize the Soup Functionality
   • SoupOpen   ' Open any existing or none existing Soup
   • SoupClose   ' Close the Soup

// Save Single Variables

   • SoupLoadDBL("name", var) 'Load a single variable Double
   • SoupSaveDBL("name", var) 'Save a single variable Double
   • SoupLoadINT("name", var) 'Load a single variable Int32 or Int64
   • SoupSaveINT("name", var) 'Save a single variable Int32 or Int64
   • SoupLoadSTR("name", var) 'Load a single variable String
   • SoupSaveSTR("name", var) 'Save a single variable String

     Note: The SoupSaveSTR and SoupLoadSTR functions can also be used to save and load a data block, with a maximum size of up to 285MB.

// Save Variable Arrays

   • SoupSaveSTR_ARR("name", var) 'Save a String Array up to 5 dimensions    
   • SoupSaveINT_ARR("name", var) 'Save a Int Array up to 5 dimensions    
   • SoupSaveDBL_ARR("name", var)      'Save a Double Array up to 5 dimensions    
   • SoupLoadSTR_ARR("name", var) 'Load a Sting Array up to 5 dimensions    
   • SoupLoadINT_ARR("name", var) 'Load a Int Array up to 5 dimensions    
   • SoupLoadDBL_ARR("name", var) 'Load a Double Array up to 5 dimensions    

// Additional Functions:

   • SoupDeleteVAR("name")             'Delete a Variable or array out of the soup          
   • SoupCountAllVARs 'Count all Variables
   • SoupVarExist("name") 'Does a Variable Name exist?


It's also possible to add a function lists all existing variable names are inside a Soup.
The soup SoupTool_V2 Demo also creates a an Int array with 500.000 Int64 elements. The corresponding Soup file owns about 6MB for the 500.000 Int64 elements.


Again I'm happy to see Sjouke working on this stuff.

[(X)]

Well done Roger!

At first glance, it seems you have created a better Data I/O scheme than the Variant Data Type which can't handle the Int64 data type!