NAND flash and JFFS2

[Rutger Hofman-2]

(Sorry if you receive this twice, my bad)

There is a steady flow of questions on the list on NAND flash and/or
JFF2. I think our group will start to do the work for that (if the boss
agrees :-) ).

The mailing list has on 14th jul 2008:
Andrew Lunn wrote:

> On Mon, Jul 14, 2008 at 03:34:49PM +0800, Fisher ZHU wrote:
>> > NAND flash is so prevalent!
>> > Will ECOS support NAND flash later?
>
> Some NAND flashes are supported, but at the moment this is no central
> support.
>
> It needs somebody to write the necessary code for all the checksum and
> block management, preferably in a way which is compatible with Linux.
>
> Feel free to submit a patch...
>
>      Andrew

And in april 2007:
Jonathan Larmour wrote:
> NAND support is not available in eCosPro either yet, although it's
> something we want to do and have looked very closely at doing already. The
> main bit missing is an equivalent of Linux's MTD layer which does bad block
> management, organises OOB data, etc. There's actually quite a lot of detail
> once you start looking at it (especially since OOB layout differs between
> some NAND parts).
>
> Jifl

And in Aug 2007:
Gary Thomas wrote:
> FYI, I looked into adding NAND+JFFS2 support some time ago and
> gave up after spending about 3 weeks trying to get it going...
> It might be easier now (the NAND stuff in JFFS2 has changed
> quite a bit, but you'll have to re-port/import the changes
> into eCos as our version is quite ancient)

For our project, we are migrating to NAND flash. We use JFFS2 currently
(on a flash-v2 branch merged into trunk). I think we have to bite the
bullet and do the NAND flash port. Well, I'm not much versed in flash
stuff, so some guidance or pointers are very welcome.

Is this the right way to go:

  = for our own system make/find a NAND flash device driver (we are
switching from XScale to BlackFin BF547 and BF537, not least because of
Marvell's defensive attitude in releasing its documentation). (Yes, we
base our work on the eCos BlackFin port by Andre Liesk, University of
Chemnitz.)

  = do I understand correctly that recent MDT/JFFS2 does the necessary
work to have good support for generic NAND flash?

  = so, then we must merge in a recent version of upstream MDT/JFFS2.
The eCos JFFS2 ChangeLog has this as its most recent entries on MTD:
     > 2005-07-30  Andrew Lunn  <andrew.lunn@...>
     >
     >         * Merge to public MTD

     > 2005-01-22  Andrew Lunn  <andrew.lunn@...>
     >
     >         * Merge from public MTD.

    How complicated is this going to be?
     - I guess upstream JFFS2 is to do with linux-mtd git. I'll have to
get familiar with that from the ground up.
     - Would it be as easy as
         git diff -date 2005-07-30 -date now | patch eCos JFF2?
       My guess is, not at all. How can I find out at what versions of
git and eCos CVS the various branches up/down occurred?

  = Or should we switch to YAFFS? It seems to have been ported to eCos,
and it claims it is really, really easy to port anyway.

Rutger Hofman
VU Amsterdam
http://www.rfidguardian.org

--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[Jonathan Larmour]

Rutger Hofman wrote:

> Is this the right way to go:
>
>  = for our own system make/find a NAND flash device driver (we are
> switching from XScale to BlackFin BF547 and BF537, not least because of
> Marvell's defensive attitude in releasing its documentation). (Yes, we
> base our work on the eCos BlackFin port by Andre Liesk, University of
> Chemnitz.)
>
>  = do I understand correctly that recent MDT/JFFS2 does the necessary
> work to have good support for generic NAND flash?

Yes and no. Certainly JFFS2 needs updating from upstream as a very first stage.

Linux MTD is fully GPL'd. As such it isn't very useful to eCos in practice
given the eCos licence is written with a special exception to avoid forcing
people to release all their application code. Use of vanilla GPL code in
eCos would oblige everyone to provide all code, including their application.

Instead eCos needs its own NAND Flash layer to perform the same sort of
things. I've included some excerpts from an internal proposal I wrote at
the bottom of this mail to give some pointers as to what is required. At
worst this should help anyone thinking about implementing it to see what's
needed so that it can be done The Right Way :-).

>  = so, then we must merge in a recent version of upstream MDT/JFFS2.
> The eCos JFFS2 ChangeLog has this as its most recent entries on MTD:
>     > 2005-07-30  Andrew Lunn  <andrew.lunn@...>
>     >
>     >         * Merge to public MTD
>
>     > 2005-01-22  Andrew Lunn  <andrew.lunn@...>
>     >
>     >         * Merge from public MTD.
>
>    How complicated is this going to be?
>     - I guess upstream JFFS2 is to do with linux-mtd git. I'll have to
> get familiar with that from the ground up.
>     - Would it be as easy as
>         git diff -date 2005-07-30 -date now | patch eCos JFF2?
>       My guess is, not at all. How can I find out at what versions of
> git and eCos CVS the various branches up/down occurred?

Not quite. If you look at http://git.infradead.org/ you'll see under
users/dwmw2 that there is a jffs2-ecos.git and jffs2-ecos-core.git
repository. The eCos parts need bringing up-to-date with the changes in the
core JFFS2 files, and bringing over any bug fixes and improvements from the
similar code in the Linux equivalents.

It's on my todo list, but then it's been on my to-do list for at least the
last year if not two, and I don't see any chance of that changing imminently.

>  = Or should we switch to YAFFS? It seems to have been ported to eCos,
> and it claims it is really, really easy to port anyway.

That is also fully GPL'd. I'm also not entirely sure the eCos port is
anything other than vapourware either, unfortunately.

Here's the proposal document I referred to:

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Background
----------

First some background, JFFS2 does work with NAND on Linux. However the
majority of the real _work_ for NAND support is done in the MTD layer. That
layer, is very associated with JFFS2, and lives in the same source
repository and everything. However it is fully GPLd. And so to all intents
and purposes, for us it is unusable.

NAND differences
----------------

Why so much effort? NAND is very different from NOR flash really.

- Access is indirect, and via a special write buffer. Access to the buffer
is fast, but then you have to wait for the operation to complete. Although
NAND erases are faster than NOR erases.
- Page sizes are small.
- It is inherently unreliable. Bad blocks are common. So common that
manufacturers often have a factory-set bad block map.
- Failures in use are also common, so NAND has a "spare area" for
out-of-band (OOB) data. This is then used for ECCs. Some devices have
hardware ECC generation, some don't. Other OOB data can be used by higher
layers, e.g. for filesystem metadata.
- You cannot write to a sector over and over again, clearing bits, like you
can with NOR.
- It does not erase to all 1s.
- It can be bigger. So much bigger that it can expose lack of scalability.
This is a problem in JFFS2, and has only partially been addressed in recent
sources. Nothing we can do about that.

JFFS2 itself
------------

JFFS2 does need to behave very differently if it's layered over NAND. Not
everything is abstracted in the MTD layer. One particular aspect is write
buffering: you cannot write a NAND flash page as many times as you want:
you are only allowed to write 1-3 times (depending on the manufacturer),
including to the OOB area. The eCos JFFS2 port, being for NOR flash, has no
support for write buffering, and the Linux support is quite dependent on
Linux kernel inode and VFS knowledge. That's unfortunate as that will
require awkward porting.

JFFS2 also stores its own fs metadata information in the OOB data, such as
clean markers. Hopefully the existing Linux code will port quite easily.

Almost all the necessary code to port is what is in Linux's wbuf.c.

There are other more minor dependencies in various other places in JFFS2,
often to do with things like (flash) page alignment. Hopefully this should
in general work, but care is required.

The MTD substitute - the eCos NAND layer
----------------------------------------

Here's the functionality the eCos NAND layer (substituting for MTD) must be
able to cover:

* bad block management
  - Whether factory-marked in the flash, user marked in flash, or user
provided in some other way

* software ECC generation
  - if no hardware ECC available

* hardware ECC management
  - device dependent clearly, but we must have infrastructure support.

* OOB layout
  - due to hardware ECC, OOB layout is not fixed and varies between flash
parts. Therefore this has to be managed and abstracted for upper layers
such as JFFS2 (which needs to know the correct OOB locations for cleanmarkers)

* provide a well-defined interface to all the above to higher layers
 - Clearly it would be desirable if the eCos NAND layer is separate from
JFFS2 (just like the real MTD is), and so can be used directly or from
other filesystems than JFFS2, rather than being tied to it. Instead it
would be a component of the generic flash layer.

* provide a well-defined interface for all the above to underlying flash
drivers.
 - probably taking inspiration from the MTD API. Although we can't just
copy code obviously!
http://www.linux-mtd.infradead.org/tech/mtdnand/book1.html is very helpful
in this regard.
-=-=-=-=-=-=-=-=-=-=-=-=-=-

Jifl
--
eCosCentric Limited      http://www.eCosCentric.com/     The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.       Tel: [phone]
Registered in England and Wales: Reg No 4422071.
------["Si fractum non sit, noli id reficere"]------       Opinions==mine

--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[Jürgen Lambrecht]

Jonathan Larmour wrote:
> Rutger Hofman wrote:
>  
>>  = Or should we switch to YAFFS? It seems to have been ported to eCos,
>> and it claims it is really, really easy to port anyway.
>>    
> That is also fully GPL'd. I'm also not entirely sure the eCos port is
> anything other than vapourware either, unfortunately.
>  
We use PEEDI debuggers from Ronetix, and on their web site they say
"Support for additional drivers for MMC/SD, NAND Flash, YAFFS2, USB
slave available from Ronetix upon request".
I requested them, and got them. It is done with a separate repository.
We merged the code in our CVS, some fixes in the generic driver
(/defs/flash/..) (we followed the Open NAND Flash Interface 2.0 spec),
some fixes in the lowlevel driver (devs/arm/..), and now it works. The
YAFFS sources were OK, but maybe not the latest versions.
We have an ARM9 board with Micron nand flash.

You only have to pay for YAFFS if you use it in a commercial product.
The code is freely accessible via CVS to play with.

I guess I can give you our code if you need it.
Let me know - the coming 3 weeks I'm on holidays, and will only read
sporadically my mail.

Kind regards,
Jürgen


--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[Jonathan Larmour]

Jürgen Lambrecht wrote:

> Jonathan Larmour wrote:
>> Rutger Hofman wrote:
>>  
>>>  = Or should we switch to YAFFS? It seems to have been ported to eCos,
>>> and it claims it is really, really easy to port anyway.
>>>    
>> That is also fully GPL'd. I'm also not entirely sure the eCos port is
>> anything other than vapourware either, unfortunately.
>>  
> We use PEEDI debuggers from Ronetix, and on their web site they say
> "Support for additional drivers for MMC/SD, NAND Flash, YAFFS2, USB
> slave available from Ronetix upon request".
> I requested them, and got them. It is done with a separate repository.

Ah that's comforting. Was the port itself by Ronetix or Aleph One?

> We merged the code in our CVS, some fixes in the generic driver
> (/defs/flash/..) (we followed the Open NAND Flash Interface 2.0 spec),
> some fixes in the lowlevel driver (devs/arm/..), and now it works. The
> YAFFS sources were OK, but maybe not the latest versions.
> We have an ARM9 board with Micron nand flash.
>
> You only have to pay for YAFFS if you use it in a commercial product.
> The code is freely accessible via CVS to play with.

That's not quite the case. The code is GPL'd which means it is free (and
Free) for any use, including commercial use, but under the terms of the
GPL. When used with eCos, this means you must be prepared to supply full
source code for every part of eCos code used, including HAL and drivers,
_and_ your application.

Anyone (commercial companies or otherwise) who wants to avoid the GPL
obligations can pay for a different licence. Obviously that's not an option
for eCos.

> I guess I can give you our code if you need it.

We can't incorporate it into the master repository, but (assuming what
you've got is the GPL'd version!) feel free to post it for others to
benefit from, if they're happy to accept the GPL terms.

Thanks,

Jifl
--
eCosCentric Limited      http://www.eCosCentric.com/     The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.       Tel: [phone]
Registered in England and Wales: Reg No 4422071.
------["Si fractum non sit, noli id reficere"]------       Opinions==mine

--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[Rutger Hofman-2]

Jürgen Lambrecht wrote:

> Jonathan Larmour wrote:
>> Rutger Hofman wrote:
>>  
>>>  = Or should we switch to YAFFS? It seems to have been ported to eCos,
>>> and it claims it is really, really easy to port anyway.
>>>    
>> That is also fully GPL'd. I'm also not entirely sure the eCos port is
>> anything other than vapourware either, unfortunately.
>>  
> We use PEEDI debuggers from Ronetix, and on their web site they say
> "Support for additional drivers for MMC/SD, NAND Flash, YAFFS2, USB
> slave available from Ronetix upon request".
> I requested them, and got them. It is done with a separate repository.
> We merged the code in our CVS, some fixes in the generic driver
> (/defs/flash/..) (we followed the Open NAND Flash Interface 2.0 spec),
> some fixes in the lowlevel driver (devs/arm/..), and now it works. The
> YAFFS sources were OK, but maybe not the latest versions.
> We have an ARM9 board with Micron nand flash.
>
> You only have to pay for YAFFS if you use it in a commercial product.
> The code is freely accessible via CVS to play with.
>
> I guess I can give you our code if you need it.
> Let me know - the coming 3 weeks I'm on holidays, and will only read
> sporadically my mail.

Well, yes, I would really like to see it. Maybe to use it, maybe as
inspiration for the eCos NAND stuff. If you don't want to post it on the
list, maybe you can send it to me privately?

Thanks in advance,

Rutger Hofman
VU Amsterdam

--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[Rutger Hofman-2]

Good evening Jurgen,

do you think there is a chance that we can have your YAFFS-on-eCos code?
If not, please let me know; then I think a good option for our project
is that I will try to also make it (and make the interface available to
eCos).

Thanks,

Rutger Hofman
VU Amsterdam

Rutger Hofman wrote:

> Jürgen Lambrecht wrote:
>> Jonathan Larmour wrote:
>>> Rutger Hofman wrote:
>>>  
>>>>  = Or should we switch to YAFFS? It seems to have been ported to eCos,
>>>> and it claims it is really, really easy to port anyway.
>>>>    
>>> That is also fully GPL'd. I'm also not entirely sure the eCos port is
>>> anything other than vapourware either, unfortunately.
>>>  
>> We use PEEDI debuggers from Ronetix, and on their web site they say
>> "Support for additional drivers for MMC/SD, NAND Flash, YAFFS2, USB
>> slave available from Ronetix upon request".
>> I requested them, and got them. It is done with a separate repository.
>> We merged the code in our CVS, some fixes in the generic driver
>> (/defs/flash/..) (we followed the Open NAND Flash Interface 2.0 spec),
>> some fixes in the lowlevel driver (devs/arm/..), and now it works. The
>> YAFFS sources were OK, but maybe not the latest versions.
>> We have an ARM9 board with Micron nand flash.
>>
>> You only have to pay for YAFFS if you use it in a commercial product.
>> The code is freely accessible via CVS to play with.
>>
>> I guess I can give you our code if you need it.
>> Let me know - the coming 3 weeks I'm on holidays, and will only read
>> sporadically my mail.
>
> Well, yes, I would really like to see it. Maybe to use it, maybe as
> inspiration for the eCos NAND stuff. If you don't want to post it on the
> list, maybe you can send it to me privately?
>
> Thanks in advance,
>
> Rutger Hofman
> VU Amsterdam

--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss