SPI Bus in Python

[Tisham Dhar]

Hi all,

I have been experimenting with the SPI bus in Python a bit and debugging it with a logic analyzer. Looks like due to the software SPI nature of the thing there is some start-up delay. e.g. if I transfer 0x80, 0xDA the data on the bus is actually 0x00, 0xDA, while if I pad by xfer buffer and transmit [0x00, 0x80, 0xDA] the missing bit appears and we have the right data, however the IC I am communicating to is watching for the first bit to perform a register read or write, with the wrong data on the bus I cannot read anything back.

I am attaching a full code file here for reference:
https://gist.github.com/whatnick/dc2c0b9429e26b8c7be0030f040d9c12

Here is the datasheet for the SPI Slave IC: http://ww1.microchip.com/downloads/en/DeviceDoc/Atmel-46004-SE-M90E36A-Datasheet.pdf

Regards,

Tisham.

[Maximilian Gerhardt]

@Tisham-Dhar You are running an old firmware in which the first bit of the first byte in a SPI transfer is corrupted, 50% of the time. Upgrade your firmware using oupgrade -l. See https://github.com/OnionIoT/spi-gpio-driver/issues/4

However in the newest version there's a problem with receiving data, though I didn't further investigate it..

[Tisham Dhar]

Hi Maximilian,

I upgraded my firmware and changed my code to read bytes by doing spi.xfer([0x80,0xD9,0x00,0x00]), the first 2 bytes are Read flag + 15bit register address, the last 2 bits are placeholder bits to keep clock going and capture output on the SPI bus.

Using logic analyser I can see correct data on the bus. However I am unsure how data gets packed into the xfer receive buffer, since this does not have correct values that I can map to what I see on the bus. I am expecting [0x00,0x00,0x00,0x47]. Please see logic analyser trace screenshot, instead I am getting [0x00,0x70, 0xFE, 0x87]. I don't see a reasonable fix other that fuzzing the SPI bus to see which data sent maps to which read bytes in the device.

Latest Python ATM90E36 Driver is here: https://gist.github.com/de3350164c68dd82e4e5e3154f57b113

[Maximilian Gerhardt]

Yep, it's broken. Just broken. Just as I, @1v3ry from the github issue and you noticed. @Lazar-Demin, could you investigate this?

[Lazar Demin]

@Maximilian-Gerhardt will do, we'll try to make some time this week

[Lazar Demin]

@Tisham-Dhar @Maximilian-Gerhardt Ok, so we spent some time looking into this.

Short Story

Instead of using spi.xfer() :
Use spi.writebytes() to write your 2 bytes, and then spi.readbytes() to read 2 bytes.

Long Story

I've tested the above with an MCP3008 and have been able to write and then read the values I see with a logic analyzer.
I've also seen that using spi.xfer like you mentioned, writing 2 bytes, and then leaving 2 bytes empty as placeholders to read data, does not work, and that I get 4 bytes of (garbage) data back.

Why xfer doesn't work involves a few moving parts, the major ones being the SPI kernel driver and python-spidev.

SPI Kernel Driver: As you probably know, we introduced a patch to work around the MT7688's hardware limitations with full-duplex transmissions. This patch allows for write-then-read behaviour in consecutive bytes, meaning, in a single transmission, we can write a byte, and then read the following byte and so on (without disabling the clock or deasserting the CS). But to switch from reading to writing, the buffer for writing needs to be null, and this depends on how the ioctl call from userspace is done. Which brings us to...

Python spidev Module: I assume the original author of the module designed it to be as generic as possible. If you look at the way the xfer function is implemented in C (lines 317-355), you'll see that a txbuf (transmission buffer) and an rxbuf (receiving buffer) of the same length are used in the ioctl call. This length is determined by the length of the list passed in from Python.
So in the case above, spi.xfer([0x80,0xD9,0x00,0x00]) will create a txbuf with a length of 4 populated with 0x80,0xD9,0x00,0x00 and an empty rxbuf of length 4. When the ioctl call is made on line 355, the kernel driver will then go ahead and write the 4 bytes (remember the kernel driver will write data unless the txbuffer data is null), and then return whatever data happened to be in the malloc-ed rxbuf.

An Idea for a Fix

While the solution of using spi.writebytes() to write and then spi.readbytes() to read will work, it would be nice to have an xfer function that keeps the clock going and CS asserted while doing the write-then-read.

This can be accomplished by extending python-spidev to have an xfer3 function:

It would look like this: values = spi.xfer3([<list of bytes to write>], <number of bytes to read>)
so in the case from the previous post: values = spi.xfer3([0x80,0xD9], 2)

Underneath in C, the function would be similar to lines 272-376 in the current implementation (ie it would use a spi_ioc_transfer struct pointer) but with a few key differences:

• the malloc-ed length would be based on the length of the input list AND the number of bytes to be read
• there would be a loop to populate xferptr elements with the bytes that should be written - along with nothing in the rx_buf element
• then there would be another loop to populate xferptr elements with an element from the rxbuf buffer, and nothing in the tx_buf element

That way we would end up with an xfrptr list that has 4 elements, the first two have single bytes in the tx_buf element and nothing in the rx_buf element, and then the last two would have nothing in the tx_buf element, but would have a place in memory to write received data.

This implementation would work in tandem with the kernel driver and likely achieve the behaviour you're looking for.

[Tisham Dhar]

@Lazar-Demin I have tried this but in my particular case the delay between writebytes and read bytes was too long and the target ic was coming out waiting for read clocks. Will give it another shot. Give readbytes uses xfer internally not sure how my mileage will vary.

[Tisham Dhar]

@Lazar-Demin As I was seeing previously writebytes followed by readbytes causes too much delay between the read and write transaction and de-asserts chip-select causing loss of the read result. I assume keeping the clock and cs going and populating rx_buffer during the write should be a good solution, actually I belive xfer should be modified to populate rx_buffer anyway since a lot of SPI devices have an echo functionality where the MISO echoes MOSI to tell the driver that the chip is acknowledging data coming in.

My suggestion:

Replace this line with read equivalent size of the xfer or xfer2 write buffer so that data read in from the kernel driver actually reflects whatever the SPI slave is placing on the MISO during the write. Passing read length of 0 seems like a waste. Simply pass it the length of the write buffer, so that we get actual data rather than malloc noise.

Also not sure the purpose of this line, why should the driver populate xfer struct with read_buffer data ? As long is data is read correctly during the full-duplex xfer we should be fine.

Another read
A third read makes the full duplex limitation and need for mediatek special xfer3 clear. So the ioctl explicitly needs to null the xfer tx buffer pointer during the 2 read bytes for this to work.

xfer3 would a possible solution once I get the onion toolchain going and can actually compile the python-spidev. Any assistance and mods from your side to implement the patch would be awesome.

[Lazar Demin]

@Tisham-Dhar Yeah, the de-assertion of the CS line complicates things.

Regarding setting up the Toolchain, take a look at the instructions in our Build System ReadMe. It will likely take the least amount of time if you use the Docker method on a Linux system.

[Tisham Dhar]

@Lazar-Demin the docker make takes a tonne of time and crashes and only seems to work 1/2 decent on a native Linux host instead of a VM on Windows host.

Would you have steps to compile a custom spidev outside of the feeds/onion/python-spidev ? This would be really useful in getting more contributions. I believe the toolchain is the biggest hurdle.

Perhaps run a CI which regularly builds the toolchain and docker snapshots for easier pull.

[Lazar Demin]

@Tisham-Dhar Yeah, we use a pretty powerful Linux server to build the firmware. You'll notice in our Build System readme that we recommend running Docker on a Linux sytem.

Don't really have instructions to build python-spidev outside of the build system. The CI is a good idea, we'll look into it and see how we can fit that into our road map.

[Tisham Dhar]

@Lazar-Demin I successfully staged and built python-spidev in isolation like this:

make package/python-spidev/{clean,compile,install} V=s

With a very handy tip from OpenWRT here.

This does not take tonnes of time compared to "make world" and is super useful for focusing my contribution. I will fork your python-spidev, patch it with xfer3 , update git link in feed to point to my fork and hack it till it works.

Anyone else is welcome to help me out and join the fun @Maximilian-Gerhardt

Regards,

[Lazar Demin]

@Tisham-Dhar Ah I misunderstood what you meant by "compile a custom spidev outside of the feeds/onion/python-spidev"
Yes, what you're doing is a good idea.
I would also suggest looking into the USE_SOURCE_DIR flag, it allows you to use/compile a local copy of the package source code. More info here.

[Tisham Dhar]

I have started a fork an have been making Python signature commits to it. The core xfer3 logic is currently a TODO. If @Lazar-Demin can review and suggest an implementation it will be much appreciated.

https://github.com/whatnick/python-spidev

[Lazar Demin]

@Tisham-Dhar the core logic can follow this function here: https://github.com/OnionIoT/omega2-lmic-lorawan/blob/master/native-spi.cpp#L110

Essentially it does two distinct transfers, one for writing and one immediately afterward for reading.

[Lazar Demin]

EDIT Sep 22, 2023: Updating below commands so Python3 version of spidev module is installed, as Python2 is now legacy

@Tisham-Dhar I've gone ahead and implemented a working half-duplex transmission function! Version 4.0.1 of python3-spidev features a new xfer3 function.

Getting the Latest Version of python3-spidev

If you don't have it installed already:

opkg update
opkg install python3-light python3-spidev

If you have it installed, you'll need to upgrade:

opkg update
opkg upgrade python3-spidev

To use the new function:

values = spi.xfer3([<list of bytes to write>], <number of bytes to read>)

This will write all of the bytes from the list in the first argument and will then immediately read the number of bytes specified by the second argument.

The function will return a list of the bytes that were read.

An Example

Instead of using spi.xfer([0x80,0xD9,0x00,0x00])

Try values = spi.xfer3([0x80,0xD9], 2)

More Details

More details can be found in our python-spidev github repo readme

Let me know how it goes!

[Juan Pablo Jimenez]

@Lazar-Demin Is this still just for python 2??? is there a python 3 version??

[Lazar Demin]

@Juan-Pablo-Jimenez A python3 version is now available as well! See https://github.com/OnionIoT/python-spidev#installation-on-omega2

[Pablo Fonovich]

@Lazar-Demin I cannot install version for python3.. it says "Unknown package 'python3-spidev'"

[Lazar Demin]

@Pablo-Fonovich Which firmware version are you running?
I ask because python3-spidev is published in the new package repo. Try upgrading your firmware to v0.3.0 or higher and trying again.