1. Welcome to our site! Electro Tech is an online community (with over 100,000 members) who enjoy talking about and building electronic circuits, projects and gadgets. To participate you need to register. Registration is free. Click here to register now.

Decoding and graphing BTc encoded data

Discussion in 'General Electronics Chat' started by vsmGuy, Aug 19, 2009.

  1. vsmGuy

    vsmGuy New Member

    Joined:
    Jun 25, 2006
    Messages:
    54
    Likes:
    1
    I am amazed by RB's work, especially the BTc encoder!

    I have been meaning to decode and graph BTc encoded data (arrays generated by his encoder) on a COMPUTER.

    I have modelled a RC filter using PERL as this allows quick prototyping and, unfortunately, the kind of graphs I get is not exact (maybe similar to my un trained eye, except at extreme ends).

    Also unlike the BTc encoder drawn waveform, the waveform I see from my data starts to oscillate between the power rail and ground after some time.

    Here is how I model the RC filter :

    Code (text):
    V(t) = V_FINAL + (V_INITIAL - V_FINAL) * (e ** (-t / RC ));
    where,
    VFINAL will be 0V if bit to be sent is low. It will be VCC if bit to be sent is high
     
    The value of RC is the value suggested by RB's application :

    R = 2966E
    C = 0.22uF

    Hence, in my code I have

    Code (text):
    my $RC = ( 2966 ) * ( 0.22 / ( 10 ** 6 ));
    This is exactly what the subroutine "V" in my code returns:

    Code (text):
    my $samplingFreq = 11000; # In Hz
    my $samplingPeriod = ( 1 / $samplingFreq ); # In seconds

    my $e = 2.7186362391351841722168029758227;

    my $RC = ( 2966 ) * ( 0.22 / ( 10 ** 6 ));

    my $numberOfSamples = 17464;

    # Returns V(t) = V_FINAL + (V_INITIAL - V_FINAL) * (e ** (-t / RC ));
    # VFINAL will be 0V if bit to be sent is low. It will be VCC if bit to be sent is high

    my $OUTPUT_AMPLITUDE = 5;

    sub V
    {
        my ( $t, $BIT_TO_BE_SENT,  $V_INITIAL ) = @_;
       
        my $V_FINAL = $OUTPUT_AMPLITUDE * $BIT_TO_BE_SENT;
       
        $t *= -1;
       
        return ( $V_FINAL + (( $V_INITIAL - $V_FINAL ) * ( $e ** ( $t / $RC )))); # Models rise as well as decay
    }
     
    I put RB's encoded data in an array called 'sound_data' and I have a loop that iterates through it a byte (element) at a time (outer for loop), extracting a bit for every time period (inverse of the sampling frequency) from the MSB to the LSB (inner for loop) before progressing to the next bit.

    This loop writes out the time and amplitude in comma separated format that I later load into Excel and graph the data, and try and match it with the graph shown by RB's Encoder application.

    Code (text):
    my $V = 0;

    for( my $dx = 0; $dx < @sound_data; ++$dx )
    {
        my $t = (8 * $dx) * $samplingPeriod;

        #for(my $bit = 1; $bit <= ( 2 ** 7 ); $bit <<= 1)
        for(my $bit = ( 2 ** 7 ); $bit > 0 ; $bit >>= 1)
        {
            $V = V( $t,  (( $bit & $sound_data[$dx] ) ? 1 : 0 ), $V);
       
            printf "%f, %f\n", $t, $V;
           
            $t += $samplingPeriod;
        }
    }
    My final target is to create WAV file from the encoded data (arrays of data).

    This should be easy to do once I have a 1:1 match of my generated graph with RB's graph. .. the place where I need some help right now :confused:

    Whole package up for grabs at :

    https://sites.google.com/site/vimalsshankar/btc

    I also attached images of "What it should be"(RB's BTC graph) and "What It is" (What Excel shows my data as)

    As you can see, my data shows that my filter starts osciallting widly just after a few samples and does not follow RB's BTC waveform as shown as all! :eek:

    Attached Files:

    Last edited: Aug 19, 2009
  2. Mr RB

    Mr RB New Member

    Joined:
    Jul 22, 2008
    Messages:
    4,717
    Likes:
    188
    Location:
    Out there
    Hi vsmGuy, and thanks for the compliments with my BTc Encoder. :)

    Decoding BTc 1bit sound back to 8bit wave is not easy at all. I've thought about it for a lot of years but there is really no easy way, that's why i have never included BTc->Wave decoding in any version of the Encoder. I am very interested in what you are doing and if it can lead to a successful decoding system.

    The problem is that the BTc encoding of the wave->1bit data encoding is very lossy. For every sound sample the encoder is generating both a 0 and a 1 bit, (based on the RC curve) then choosing the "predicted" bit that is closest to the original wave. The encoding algoithms are shown on my
    Roman Black's BTc 1bit Sound Encoding Algorithms
    (BTc Algorithms page)

    So each BTc bit will have an error, that may be + or - to the original sample, by quite a large amount. Then the next bit will have another error that will tend to correct the previous error(s), because every new bit chooses the best option (+ or -) to get back to (closest to) the original waveform.

    This randomness of the error is extremely hard to reverse engineer being that the + or - for that bit is unknown, as were the bits before it that were also unknown. Also the predictive nature of the algorithm means there is a time issue as the BTc bit is predicted in advance of the new wave bit.

    Sorry I don't know PERL so I cant help you much there, but one thing seems an issue is that my algorithm uses the middle part of the 5v (ie the orig wave is reduced to half amplitude and then biased at 2.5v centre, so it goes from 1.25v to 3.75v only). Your first diagram seems to have a 0v-5v swing which would not be possible given that RC time constant and that bitrate. If you change that it should get you back close to the BTc waveshape. Again, check my encoding algorithms on the link above, reproducing the BTc waveform from the BTc data should not be that hard.

    I really don't think you can reverse encode the BTc data back to a wave based on a per-sample system for the reasons I stated above. The only way I think it CAN work is to do it the same way your ear does, by a low pass averaging of the total waveform. By averaging each sample with the samples before AND after (remember it's predictive) you should be able to remove most of the PCM noise and get a re-coded (not decoded) wave that is hopefully something like the original wave.

    Hope that helps.
  3. vsmGuy

    vsmGuy New Member

    Joined:
    Jun 25, 2006
    Messages:
    54
    Likes:
    1
    Thanks a lot Roman for the detailed answer!

    Well what I am trying to do is simple really - it should mimic the waveform that your encoder application plays when one clicks on "Play BTC Sound".

    I am not trying to do anything else - I need to hear what the sample might sound like out of the uC output pins!

    What I want to do is to create offset pointers into sound. For example, say I record :

    "Counting up. Now going down"

    As you can see, we have a lot of data that is common :

    "Count" "ing"

    "go" "ing"

    So if I can save a pointer to ofsset and lenmgth into "ing" of "Count" "ing", I just need to store "go" and I get "go" "ing" for free!

    Something like that :)

    I noticed you also have a pointer creation facility in your tool - is this the same?

    To summrize - my tool will allow you and anyone else to play the NTc encoded binary output WITHOUT actual hardware :)

    Is there something available to do that right now?
  4. Mr RB

    Mr RB New Member

    Joined:
    Jul 22, 2008
    Messages:
    4,717
    Likes:
    188
    Location:
    Out there
    Hi, re the pointers, yes the new version of BTc Encoder v3.0 handles sound pointers for you, this was a feature people had asked for that allows you to join sounds into one "library" or just to play separate parts of sounds as you said above. It really makes the system useful for makng a PIC talk etc as it can speak any word from within the sound library.

    As for playing the BTc encoded 1bit sound, my software already does that (during/after encoding), there's a button for it.

    If you want to play other 1bit BTc data as a wave file, you can simulate the PIC hardware playback of the 1bit stream. Sorry there is no tool available that currently does playback simulation (apart from actual playback in real hardware).

    If you want to code up a simulator, you can reverse the encoding algorithm. From my algorithm page; here is the procedure for encoding 1bit sound at BTc8;

    Code (text):

    variables;
        NewSample = the new audio sample, scaled to 1/4-3/4 Vcc range
        LastVoltage = the cap voltage (audio output) from the last bit
        NewVoltage = the cap voltage (audio output) after we encode this new bit
        NewBit = the newly encoded BTc bit

    encoding;
    1. generate a 1bit;
        distance = (Vcc - LastVoltage) / 8
        Voltage1 = LastVoltage + distance
    2. generate a 0bit;
        distance = (LastVoltage - Gnd) / 8
        Voltage0 = LastVoltage - distance
    3. Find which bit is closest to the desired audio sample;
        if Voltage1 is closest to NewSample then; NewBit =1, and NewVoltage = Voltage1
        if Voltage0 is closest to NewSample then; NewBit =0, and NewVoltage = Voltage0
    4. Save the cap voltage and repeat for the next audio sample;
        LastVoltage = NewVoltage
        Repeat
     
    So to simulate playback from a PIC to a RC filter, you can basically reverse the encoding algoithm. This will simulate the analog voltage on the cap in the RC filter of the playback hardware. The simulation procedure would look like this;

    Code (text):

    variables;
        TheBit = the BTc bit (from your data file)
        LastVoltage = the cap voltage (audio output) from the last bit
        NewVoltage = the cap voltage (audio output) after we simulate this new bit (this is the 0-255 "wave file" output)
       
    1. Start at middle of waveform;
        LastVoltage = 128
        NewVoltage = 128

    2. Get the next bit;
        If bit==1; NewVoltage = (LastVoltage + ((256 - LastVoltage)/8) )
        If bit==0; NewVoltage = (LastVoltage - (LastVoltage/8) )

    3. Now NewVoltage is the "wave" output sample, so save it

    4. Keep a copy of the cap voltage
        LastVoltage = NewVoltage

    5. Go back to 2, until all bits are simulated.
     
    You should be able to make that a PERL script without too much hassle. The simulation does assume you know what algorithm as used in encoding (in this case was alg1) and assumes you know what the BTc fineness value was (in this case it was BTc8).

    Because of the binary nature of the time constant it's extremely easy to simulate the RC filter resonse, in that for every bit (every sample) the voltage on the cap moves 1/8th of the way towards MAX (for a 1 bit) and 1/8th of the way toward ZERO (for a 0 bit). It's 1/8th because it was encoded at BTc8 obviously.

    The simulation code above will give you an unsigned 8bit "wave" file with each sample from 0-255, however the waveform will slightly reduced and be closer to the centre of the range since it was originally encoded from a 1/2 amplitude waveform.

    Now that simulated waveform will NOT be a perfect reproduction of the original wave due to the lossy nature of 1bit encoding. It will contain quite a lot of PCM noise "aliasing" etc that normally in hardware is reduced by low-pass filtering in the amp and speaker. If you want to play this as a wave file you can low-pass filter this farily easily to get an IDEA of what it would sound like, one technique I used was just to average every wave sample with the sample before it, this smoothes much of the PCM noise and gives an approximation of how the BTc 1bit playback will sound in actual hardware after filtering.
    • Like Like x 1
  5. vsmGuy

    vsmGuy New Member

    Joined:
    Jun 25, 2006
    Messages:
    54
    Likes:
    1
    So you mean so say, that if I have a sample that sounds :

    "Counting up. Now going down"

    Then I can access "Count", "ting", "up", "go", "down" from my hardware as pointers into a look up table? :confused:

    I did click around the set offset options in your application but found it hard to understand.

    Had I exported the data to C/ASM after setting the pointers, would have got some different kind of output?

    Could you help me out a little bit by giving on overview of this option :)

    Yes, but I am trying to figure out how to play data from middle of the array and I have not figure out the pointer option in your application yet :eek:

    This is exactly what I am trying to do :D

    Thanks a lot Mr RB!

    If I still don't get the thing to work with all the information, maybe I should dig a well and die inside it :(

    Thanks again for sharing this wondeful stuff with us!
  6. Mr RB

    Mr RB New Member

    Joined:
    Jul 22, 2008
    Messages:
    4,717
    Likes:
    188
    Location:
    Out there
    Sure, that was the main change I did for BTc Encoder v3.0.

    1. Open your wave file that says "Counting up. Now going down"

    2. Goto Edit->edit library pointers

    3. Enter the sample position (number) for the end of sound0 "Count"
    (you can click PLAY to hear the sound for sound0)

    4. goto sound1, put in the end point sample number for the end of sound1 "ing" (you can play it to test it like above)

    5. keep going until you have all sound pointers allocated.

    6. Save as .SL sound library file, this is a wave file that keeps your sound pointers (in a 1024 byte header) as well as the wave data

    7. Export the sound as .BTL BTc library file, this is the BTc 1bit sound data to play on your PIC, but also has the same sound pointers (in the same 1024 byte header).

    8. now your PIC or AVR can automatically play any of the words because it has the 1bit sound data AND all the sound pointers.

    As I said above that was the feature people wanted me to add to my 1bit sound, so that there can be multiple counds in the one file (up to 256 sounds) and the file also contains the sound pointers so your PIC or AVR playback hardware can just select "play sound0" or "play sound4" etc.

    There is more about the sound pointers and sound library files on this page;
    Official homepage of TalkBotBrain.com
    where you can also get cheap ready-made hardware to play sounds with a PIC and 1 Megabit eeprom included.

    Please let me know if/when you get the simulator coded, if you like I can put a link on my webpage so other people using 1bit sound can make use of your work too. :)
  7. vsmGuy

    vsmGuy New Member

    Joined:
    Jun 25, 2006
    Messages:
    54
    Likes:
    1
    Thanks for the walkthrough RB!

    I feel like a fool though now.

    Well, the thing is that I got the simulation working fine. The code generates the wav file and I get to hear the sound as I hear it from the uC without a LPF.

    My code is unpolished though - only meant to simulate BTC8/1bit arrays .. will this limitation be OK?

    I can work with you though to make sure it can simulate any of your algos - that would be much better :)

    The nice thing using my Perl code however is that it will run on Linux or any OS that has a Perl interpreter - guess that increases the reach of your code :)

    Will update this site with links, code and samples so you can get the code to update to your site and others can refer back when they want.

    I have a few questions for you though :

    1. The .BTL BTc library file that contains the pointers - this seems to be in a binary format? (Is the layout documented here : http://www.talkbotbrain.com/index.htm#libformat ? )

    In that case it seems that the sample length pointed to by each pointer need to be multiples of 32 samples (and the last sample can be smaller?)

    For example, I am thinking of simulating DTMF tones that can be as small as 2 - 3 bytes!

    Can I have the pointers pointing to these samples? (or will extra space be consumed in padding the 2 -3 bytes data to 4 bytes to align it to sample/chunk size?)

    2. Do we not simply bit bang the array elements out the uC pin? Then why did not my original equation fail to simulate the waveforms?

    Although now the graphs I do have a 100% match with the ones generated by your application, I am still wondering why the original equation lifted striaght out of a college textbook fails to reproduce the final waveform?

    Of course my model is wrong as evident from the fact that now I get exact matches after listening to you, but the routine in the last but one post you wrote is more of a digital approximation than an analog one?
    Last edited: Aug 23, 2009
  8. Mr RB

    Mr RB New Member

    Joined:
    Jul 22, 2008
    Messages:
    4,717
    Likes:
    188
    Location:
    Out there
    Great! Glad you got the simulator working, as i said it would be easy to code up that's the beauty of BTc the sheer simplicity of it.

    The help file .HLP included with BTc Encoder has a lot of info on sound pointers and the library header format.

    Re sample size per-sound in the BTc library file it is limited to 32 samples (4 bytes) as that is the smallest sound size. That limitation was chosen to allow a 2byte address (in PIC playback) to be able to reference 2 megasamples of sound. A 32 sample resolution is fine enough for most sounds considering that's like a millisecond or so.

    I'm not sure why your orignal PERL app didn't work, but it was trying to do the RC filter modelling the hard way using floating point. The whole benefit of my BTc system is that it only needs BINARY divisions and a few add/subtracts etc, its extremely fast and bulletproof.

    Please PM me when you have updated your page, and I will put a link on my BTc page to your page so people can access your work, or I can even put your PERL files on my page if you think your web page won't be there permanently as that might be better. And we should probably continue any development via PM's instead of on the open forum as it might irritate the moderators doing it in this thread. :)
  9. vsmGuy

    vsmGuy New Member

    Joined:
    Jun 25, 2006
    Messages:
    54
    Likes:
    1
    Hi Roman!

    I would prefer you host the files on your site as that will ensure everything is available at one place - I just cannot stop emphasizing how exciting your project is!

    Do PM me your mail address so that we can correspond without bloating this thread (You rightly pointed out that the conversation has been only between us) but I will help you post latest updates to your site.

    I do have a question though - the model of the complete decoding system that constitutes a uC and a RC filter with us taking the output across the capacitor does follow the exponential formulae with the voltage across the uC pin feeding the capacitor being either 5v or 0v at any sampling period - does it not?

    Could you please, in that case help me solve the large discrepancy between the waveforms that I am getting between the BTc waveform and the odd waveform I am getting from my script?

    :-D
  10. Mr RB

    Mr RB New Member

    Joined:
    Jul 22, 2008
    Messages:
    4,717
    Likes:
    188
    Location:
    Out there
    For the 1bit algorithm (that we have been talking about) that is correct.

    Just use the procedure i suggested above and convert that to PERL. Which I thought you already did? You said it was working ok to reproduce the graph waveform 100%.

    If you like, PM me your PERL script for the procedure I specified above and i'll try to help with it. I'm not going to debug your original PERL script because it didn't work and it was a terrible way of doing the task compared to my procedure.
    Last edited: Aug 24, 2009
  11. vsmGuy

    vsmGuy New Member

    Joined:
    Jun 25, 2006
    Messages:
    54
    Likes:
    1
    :D

    I was talking about the original script. The one based on your last discussion works 100%.

    Oh well .. I will mail you a finished version of the scrip and some supporting file soon.. gove me a day or to to make sure it is presentable :eek:

    Thanks again for all the support (and any insights into why the original script did not work at all - does it not model the BTc decoding process as in real life?)
  12. vsmGuy

    vsmGuy New Member

    Joined:
    Jun 25, 2006
    Messages:
    54
    Likes:
    1
    Hi RB!

    Latest files available here https://sites.google.com/site/vimalsshankar/btc

    I would like to add a functionality to your application :

    The BTc encoder application should output the BTc encoded data in true C array "format" : a list of bytes, ( helps to run in a C compiler as well as PERL etc )

    If you don't have time to do this yourself, I would be more than happy to get this done!

    :)
  13. Mr RB

    Mr RB New Member

    Joined:
    Jul 22, 2008
    Messages:
    4,717
    Likes:
    188
    Location:
    Out there
    Cool. I'll check out your latest files. :)

    Umm, my software already does export the BTc sound data as C bytes. See File->Export PIC MikroC file .C But if you want to add some different or extra functionality then sure please go ahead.

    I'll PM you after I check your files. :)
  14. vsmGuy

    vsmGuy New Member

    Joined:
    Jun 25, 2006
    Messages:
    54
    Likes:
    1
    :D

    Actually, the C format you export to is like this :

    Code (text):
    void sound_data1()    org (1 * 256)
    {
       asm retlw 0x56 ;
       asm retlw 0xAB ;
       asm retlw 0x2B ;
    ...
    I want it like this :

    Code (text):
    const char sound_data1[] =
    {
       0x56, 0xAB, 0x2B...
    };
  15. Mr RB

    Mr RB New Member

    Joined:
    Jul 22, 2008
    Messages:
    4,717
    Likes:
    188
    Location:
    Out there
    Ahah. I get you now. Sure, that looks handy.

    If you want to script that up I will add it with your other stuff to the BTc page. :)
  16. Mr RB

    Mr RB New Member

    Joined:
    Jul 22, 2008
    Messages:
    4,717
    Likes:
    188
    Location:
    Out there
  17. vsmGuy

    vsmGuy New Member

    Joined:
    Jun 25, 2006
    Messages:
    54
    Likes:
    1
    This is so cool :D

    No need to apologize at all. Thanks again for sharing so many nice things with the world :)

    I am really interested in the TalkBotBrain Project. If you need some work done for which you are short on time, please let me know.

Share This Page