Ambiguity between scalar value and array with shape (0,) or (1,)

[prjemian]

There exists an ambiguity in HDF5 files between scalar values and arrays with rank=1 and length of 0 or 1 item. AFAIK, we always change such items to be scalar values.

Would it be useful to add an (optional) as_scalar="true" boolean attribute to such a dataset to disambiguate? This would formalize a convention we have recognized as implicit. With the converse (as_scalar="false"), we could properly identify arrays with length of 0 or 1.

[rayosborn]

Following bitter experience reading NeXus files from different sources, the nexusformat API now reads NeXus fields exactly as they are written, but when comparing their values to other variables, they are converted to scalars if they have a shape of (1,) or (). This is a mine-field:

>>> a=np.ones(shape=(1,), dtype=int)
>>> a.shape
(1,)
>>> a==1
array([ True])
>>> b=NXfield(a)
>>> b.shape
(1,)
>>> b==1
True

I believe that someone reading this file would expect/prefer the behavior of b rather than a. This is produced by providing two different field attributes, nxdata and nxvalue. The first gives what is stored, the second what is almost certainly preferred (for example, converting byte strings to unicode).

>>> b.nxdata
array([1])
>>> b.nxvalue
1

This is, of course, one of the reasons I recommend using the nexusformat API. It normalizes the many disparate ways people write NeXus files in a way that h5py on its own cannot.

I would not recommend that we add extra attributes until someone provides a realistic use case where size-one arrays should be treated differently from scalars. If such use cases exist, we need to specify when they are valid.

[vasole]

I prefer to read what it was written in the file. If it is size 0 or size 1, I consider the writer had a good reason to do it.

For instance, a scan of 1 point or a scan of n points would logically have shape 1 in both cases and an array would be the logical thing to be expected.

If it is me (or Nexus) who expects an scalar, then it is my business to decide if I want to make the conversion from array to scalar.

[rayosborn]

The problem is that I have encountered string attributes, such as the 'signal' attribute, stored as a size-one byte array. If someone writes code such as:

>>> field.attrs['signal']=='Q'

they will not get the answer they expect. Of course, you can write to catch all the possible variants that people have used to write NeXus files (and all have been used, believe me), but the beginning user who just wants to read a file will just decide that NeXus is too much of a mess to bother with. I believe that most of them just want it to work, and couldn't care less about the niceties of size-one arrays vs scalars.

[vasole]

With string attributes is a real mess. It is one of the cases where Nexus says one thing (i.e. a string) but you encounter all the flavors (i.e. arrays of one string). Therefore in that case one converts to what is expected being more or less verbose concerning warnings.

My comment was due to the example shown above. Nexus scan rules allow to use both scalars or arrays for many datasets ("fields") and there I clearly read what is written. For the same data writer, I do not think a reader would appreciate reading different data types (scalar or array) if the data correspond to one point or to n points IF the writer wrote originally the same data type in both cases.