Developing with HTU21DF Sensor part 4.

In previous posts we have detailed

In this post after laying ground work, let us start development and write code to capture Humidity and Temperature from Sensor. Code is written in C# using “Universal Windows Platform”

Am developing each module at a time and sharing it on GITHUB @ https://github.com/gurucb/HomeAutomation. For those we need code (without github write to me, gurucb@outlook.com, I will share them code).

Advertisements

under hood of HTU21DF sensor part 3

Cyclic Redundancy Check:

To ensure any detection of raw data modification (corruption) of sensor(HTU21DF) measurement, Cyclic Redundancy Check is used. Polynomial used for CRC is

image

Above polynomial can be written as

image

taking coefficients of all terms our binary values are 1 0 0 1 1 0 0 0 1. This binary value or polynomial generator is used a divisor for CRC computations.

Compute checksum: (Best refer to Wikipedia link Smile) or there are readymade tools. But if you are interested to know below are steps..

  • Step 1: Take input bytes from Sensor (MSB + LSB as detailed in previous post). They are 16 bits longs (2 bytes). Append to them 0’s 8 to right (should be equal to length of polynomial generator). <Update> As we are stuffing 0 to right of data bytes, effectively “Left Shift” operation occurs.
  • Start from left most bit that has bit value of 1 and perform an XOR bitwise operation with polynomial generator. Skip any 0s in middle and perform operations if bit left most is 1.
  • Repeat these operations moving Polynomial generators to right, where bit value is 0 till 16 bits.
  • In Step 1, due to polynomial movement, initially stuffed 0’s will have changed to different state and that result is output of CRC computation.

As an example, in previous example measured humidity is 31827 when converted to binary is input. Using polynomial generator in an iterative manner compute CRC. Sample computations are below.

image

In this manner, receiving application can compute CRC for MSB and LSB compare with CRC byte to check if data received is not corrupted.

Post measure and performing CRC checks, formula need to applied to raw measurements to arrive at accurate values as provided in manual.

Relative Humidity Conversion:

image

Where S(RH) is Humidity Signal Output that is measured from Sensor.

Relative Humidity Conversion:

image

With this HTU21DF sensor internals are complete… As continuation to this series of blog, follow up blog will detail how to programmatically get data out Sensor using RASPBERRY PI 2 MODEL B and SPARKFUN WEATHERSHIELD that includes HTU21DF sensor…

Until next post…

–Abhyast

under hood of HTU21DF sensor part 2

Schematic view of sensor with Pins:

image

  • GND: Ground
  • VDD: Voltage for Sensor between 3.6 to 5 Volts DC
  • SCK: Serial Clock Input is used to synchronize communication between master(microcontroller) and HTU21DF sensor.
  • Data: Serial data pin is used to transfer data in and out of sensor and device.

Communication Sequence with Sensors:

  • Start Sequence: A single bit. To initiate transmission start bit has to be issued by microcontroller.
  • Stop Sequence: A single bit again to indicate completion of transmission of data (along with CRC) and if required release holding of SCK.
  • Sending Command: After start bit to indicate initiation transmission, next is to send command to sensor identified by sensor address (similar to MAC address)
    • Device Address of HTU21DF address in hex 0x40 (in Decimal system = 64)
    • Device address is 7 bits wide with last bit capturing type of operation (Read / Write)
    • Thus controller will send initiate communication to HTU21DF Sensor as below for both read and write operations

As I2C protocol specifies, Start –> Address of Sensor (With Read / Write intent)

Read Operation:

image

Write Operation:

image

HTU21DF sensor responds (as its address is 0x40) with ACK (Acknowledgement). Then controller sends command to HTU21DF sensor. Commands their Hex and Binaries are below.

image

In comment section above there are “Hold Master” and “No Hold Master”. As detailed in previous post, I2C protocol provides options where sensor may choose to control serial clock (SCK).

Hold Master ensures control of SCK is with Sensor and controller can not perform any other actions with other sensors if available but is kept in waiting state till sensor measurement is complete. On the other hand, No Hold Master implies, control is SCK is with master (controller) and controller is free to communicate with other sensor. Additionally a polling is done with read command to check if measurement is done.

HOLD MASTER is analogous to synchronous programming and NO HOLD MASTER is equal to Asynchronous programming but with polling. May be NO HOLD MASTER is good for situations where there are more sensors on board.

Post measurement, data is sent by Sensor 2 bytes MSB(Most Significant Byte) followed by LSB (Least Significant Byte) and a checksum.

Additionally on LSB, last 2 significant bits are reserved for transmitting status information. In LSB (Bit 1 is used to indicate temperature / Humidity). 0 Temperature and 1 Humidity.

Communication Flow:

Step 1: Start (Initiate) session to Sensor (Slave in this case HTU21DF)

Step 2: Issue Write intent to sensor indicated by address

image

Step 3: Sensor sends ACK to master

Step 4: Issue “Measure Humidity with HOLD” command

image

Step 5: Sensor sends ACK

Step 6: Issue read intent to sensor with address of sensor.

image

Step 7: Sensor takes control of SCK line and makes master (microcontroller) wait till measurement is complete.

Step 8: Send Data. Post measurement sensor uses 14 bits (of data) + 2 bits (status) to send measured data using SDA line as output from device. Data is divided into 2 sections Most Significant Byte (MSB) and Least Significant Byte (LSB). First sent is MSB followed by LSB.

image

In above example, relative humidity is measured as 31827 (add numbers on right column (except status bits). Status column 1 indicates measurement is humidity (0 implies temperature).

Step 9: HTU21DF sensor provides CRC8 checksum. is computed by sensor and sent as an additionally redundant byte for data verification.

In this manner communication takes place between master (microcontroller) and I2C compliant sensor like HTU21DF.

Next section we will understand formulae for RH (relative humidity), Temperature and also verify data integrity from device using checksum provided in checksum Byte.

Appreciate if readers review and correct mistakes

Till then….

Abhyast..

under hood of HTU21DF sensor part 1

There is so much to learn and so many moving parts in technology now a days, am always reminded of what I do not know, rather than what I know. Hope that clarifies my very basic knowledge and would be glad if someone corrects mistakes in this post. In this post I am trying to learn Sensors to do a side project in my company. So this is a idiots guide to sensor specifically HTU21DF sensor.

HTU21DF is a digital sensor for capturing “Humidity” and “Temperature”. Sensor provides signal in I2C (I Squared C)  format. I2C is a serial communication protocol that communicates over Two Wire Interface (TWI). In this post, let us learn about I2C / TWI Protocol.

I2C / TWI Protocol:

  • A Micro-controller will have limited number of pins.
  • But sensor that a micro-controller can serve are many but limited by pins.
  • A two wire interface bus is built around 2 lines
    • 1 SCL – Clock (Synchronizes data transfer between master & sensors)
    • 1 SDA – Data (Carries data)
  • These two wires / bus helps connect multitude of sensors to Microcontroller.

Schematically I2C bus along with sensors is something as below, in which each sensor is Slave is connected to microcontroller using two wires (bus) one for Signal Clock and other for Signal Data

image

With 7 Bit address, 128 devices (sensors+ Masters) can be added to controller and with 10 bit addressing 1024 devices can be used using serial bus. Every sensor has unique address on bus, that master uses to communicate to that sensor.

Data Protocol:

Chunk 1: 

  • S= Start Bit.
  • A0 to A6 are address bits for Sensors for data transfers
  • A7 type of operation (Read / Write) that master will perform on slave.
  • Followed by Acknowledge bit used by slave to acknowledge receipt of previous sequence of bits from master.

image

After ACK, master device hands over control of SDA link to sensor with address in Address bits.

Chunk 2:

image

Post ACK, are bits for internal registers for sensors. For example temperature  / humidity sensors will have internal registers to capture either address  / data. When detailing HTU21DF we will deal again with these internal registers.

Chunk 3:

Data being transmitted either from Master to Slave to Slave to Master depending upon RW bit set in Chunk 1 and finally ends with a Stop bit to indicate communication between Slave and Master ends.

image

Above outlines how I2C protocol works on a Two Way Interface using Serial Bus Architecture.

Next up is understanding specific sensor, HTU21DF in terms of “Address”, “Internal Registers” and “Data Format” packet.

Immediately following up with next blog that utilizes these concepts with sensor..

–Abhyast

Event Based ARchitecture

Learning and understanding any technology (aka, rampup), more abstract from design principle and technical implementation, more transfrable , reusable knowledge. Or simply put, understand product or solution to its utmost depth (bordering academics), so that knowledge thus gained is valid across technologies and stands perinial.

Am given a task to understand, compare “Complex Event Processing” engines like

  • Apache Storm
  • Amazon Kinesis
  • Azure Stream Analytics

Started with reading an excellent article on Event Driven Architecture by Brenda M. Michelson https://dl.dropboxusercontent.com/u/20315902/EventDrivenArchitectureOverview_ElementalLinks_Feb2011.pdf.

Have I experienced Brenda’s architecture, hence this blog.

In an “Event Based Architecture”, EVENT is a central abstraction and basis of everything that happens post event. EVENT is highly generalized, to include anything that happens, be it due to sensors or smart devices, shop floor machines,  business transactions (order or inventory management systems) through services and solutions or could be due envoirment. Also EVENT concept is all encompassing, anything that happens under “Universal Model” till BIG BANG can be viewed as event. Philosophically, Life is a sequence of notable, forgettable sequence or layers of events.

One implicit assumption with EVENT being base  is,  “event” occured / took place in the past, detected and captured. “EVENT BASED ARCHITECTURE” deals with what to do with an event and how to handle events when scale is necessary.

Question should not be whether events occur, but more apt questions are

  • Do we / can we capture all events that occur?
  • If yes, are they interesting and what can be learnt?
  • How to learn from event and what action (if required) to be taken?

Events are Sensed (Sensor) / Probed (Labs) / Detected , analyzed and consumed.

Event:

  • An “Event” is a notable thing that happens that could be either internal or external to scoped system. Scoped System is field of interest could be Organization or any other entity.
  • Event term is generally used to indicate “Occurance” and “Specification” of a notable thing. Specification of what of Event including “Event Header” and “Event Body” while in Occurance captures when of event.
  • Specification of Event needs to be as complete as possible for downstream applications to analyze and act on event.

Event Header: //More Generic in Nature. For all events
{
    Event ID: <Unique Identification Number>  ,
    Event Type: <Type of Event>
    Event Name: <Name of Event>
    Event Timestamp: <When was Event generated>,
    Event Origin: <Event generated by>
}

Event Body:
{
    Other Event Details:
    {
        //Specific to Scope of work.
        //Standardization across Event Generators / Consumers necessary
        Event Code:
        Event Description:
       
    }
    Event Deviation Details:
    {
        Current Reading:
        Threshold Reading:
        Min Reading:
        Max Reading:
    }
}

 Event Generators: Events are generated by source. Source is a generic abstraction to include anything from Software Application, Service, Sensors, Transmitters, Business Process. Every events may not be necessary in all cases, only interested events (like deviations from threshold) may be necessary for downstream applications, provided thresholds or deviation boundaries are well defined. Each event is evaluated and from a universal set of all events, subset of notable events are filtered to be further processed by downstream applications. As indicated, evaluation and filteration of events is based on context and type of work.

Event Channel: Messaging backbone to transports events for processing by downstream entities. Event channel may need to persist state and ensure events are durable and not lost during transit.

Event Processing: Upon receiving events, events are evaluated against processing rules (analyze phase) and actions are initiated as prescribed (act phase). Evaluation of events and actions are defined as required at event processing layer and is completely indepdent of needs of event generators. For Event processor, how to evaluate an event and what action to take is primary concern rather than who / what raised event. Similarly, Event Generators are concerned only with raising an event and event processing is a black box to them.

This decoupling is achieved best only when event generators, provide all necessary data points related to events so that event processors  are not stuck due to lack of data.

Relationship between Event Generators and Event Processors:

Event generators and Event consumers are very loosely coupled and only intermidate channel is connecting point between them.

Due to loose coupling

  • Event generators or Event Processors can be scaled out independent of each other.
  • A lag could be introduced between time event was generated and when it was processed.

Event processing patterns are of three types.

  • Simple Event Processing
  • Stream Event Processing
  • Complex Event Processing

Frankly speaking, I have not yet got grasp of Stream Event Processing in given examples, still trying to understand in my own terms what are use cases for each and explain in more detail.

Once I get my head around these three patterns of Event Processing, will put out my thoughts and ideas..

Till then..

Abhyast..

unified theory of everything

Currently am reading book “Doing Data Science” that asks reader to ponder on one of thought experiments, “How would data science differ if we had grand “UNIFIED THEORY OF EVERYTHING”

Disclaimers & Credis:

  • Neither I am expert machine learning nor do I possess intellectual or other capabilites to even grasp / internalse entire meaning and impact of such a profound question.
  • Quotes that I use here are NOT mine but are used from various books / articles / papers that I came across. And credit is given to all such authors where due.

With disclaimers and credits out of way, let me start with one quote that captures in everysense what this question poses. “Man is a microcosm of Macrocosm”.

micro|cosm

NOUN

  1. a community, place, or situation regarded as encapsulating in miniature the characteristics of something much larger:

    “the city is a microcosm of modern Malaysia”

    • humankind regarded as the representation in miniature of the universe.

      “the belief in correspondences between the Universe and Man—between microcosm and macrocosm”

macro|cosm

NOUN

  1. the whole of a complex structure, especially the world or the universe, contrasted with a small or representative part of it

Combining both definitions, “Every Man is an instance of humankind, which as a whole  is representation in miniature of universe, universe being THE WHOLE OF COMPLEX STRUCTUREimage” Or mathematically in its most simplistic terms it may be written as

 

where

M = Macrocosm , m = Microcosm and

image implies that delta (differential component) to integrate Microcosm (m) to reach level of Macrocosm (M) could be many different attributes like

  • Time: Time since Big Bang to next Big Bang
  • Behaviour
  • Abilities / capabilties
  • State of body, mind & spirit??

(integrated) become into something that may be we are still not able define in scientifc terms…. Just like in Normal distribution, if that grand unidified equation captures variation and confidence intervals in everthing that happens right from birth of a star to birth of a sub atomic particle, then there is degree of certainity in everything and may be there is no need of data science. Data science exists if there is “randomness and variability”.

Just a fun post.. to do justice to my reading book nothign more.

Do you think I am crazy.. leave comments…

Guru

“DAY 3: LEARN 3 THINGS A DAY – LEARNING 2”

Data Vault Series 1– Data Vault Overview

http://tdan.com/data-vault-series-2-data-vault-components/5155

In DV model task of data transformation, cleansing and de-duplication is moved from Source to DWH (Data Warehouse) to DWH to DM (Data Mart layer). Thus effort needed to move data from disparate source systems to centralized DWH model becomes simpler . As I understood till now

 

  • Using DV modeling simplify, expedite data modeling and loading data from source system to DW model.
  • Simply modeling to dimension models for data marts preferably by using views (for faster loading into dimension model) unless performance becomes an issue.

As detailed in previous learning post, main entities of DV modeling are

  • Hubs
  • Links
  • Satellites

For this learning I am using AdventureWorks database. Picked few tables with scripts below

CREATE TABLE [ProductCategory](
    [ProductCategoryID] [int] IDENTITY(1,1) NOT NULL,
    [Name] [dbo].[Name] NOT NULL,
     CONSTRAINT [PK_ProductCategory_ProductCategoryID] PRIMARY KEY CLUSTERED
    (
        [ProductCategoryID] ASC
    )
)

CREATE TABLE [ProductSubcategory](
    [ProductSubcategoryID] [int] IDENTITY(1,1) NOT NULL,
    [ProductCategoryID] [int] NOT NULL,
    [Name] varchar(100) NOT NULL,
    CONSTRAINT [PK_ProductSubcategory_ProductSubcategoryID] PRIMARY KEY CLUSTERED
    (
        [ProductSubcategoryID] ASC
    )
)

CREATE TABLE [Product](
    [ProductID] [int] IDENTITY(1,1) NOT NULL,
    [Name] nvarchar(100) NOT NULL,
    [ProductNumber] [nvarchar](25) NOT NULL,
    [Color] [nvarchar](15) NULL,
    [SafetyStockLevel] [smallint] NOT NULL,
    [ReorderPoint] [smallint] NOT NULL,
    [StandardCost] [money] NOT NULL,
    [ListPrice] [money] NOT NULL,
    [Size] [nvarchar](5) NULL,
    [SizeUnitMeasureCode] [nchar](3) NULL,
    [WeightUnitMeasureCode] [nchar](3) NULL,
    [Weight] [decimal](8, 2) NULL,
    [DaysToManufacture] [int] NOT NULL,
    [ProductLine] [nchar](2) NULL,
    [Class] [nchar](2) NULL,
    [Style] [nchar](2) NULL,
    [ProductSubcategoryID] [int] NULL,
    [ProductModelID] [int] NULL,
    [SellStartDate] [datetime] NOT NULL,
    [SellEndDate] [datetime] NULL,
    [DiscontinuedDate] [datetime] NULL,
    CONSTRAINT [PK_Product_ProductID] PRIMARY KEY CLUSTERED
    (
        [ProductID] ASC
    )
)

CREATE TABLE [SalesOrderHeader](
    [SalesOrderID] [int] IDENTITY(1,1) NOT NULL,
    [OrderDate] [datetime] NOT NULL ,
    [SalesOrderNumber]  NOT NULL,
    [CustomerID] [int] NOT NULL,
    [SalesPersonID] [int] NULL,
    [TerritoryID] [int] NULL,
    [BillToAddressID] [int] NOT NULL,
    [ShipToAddressID] [int] NOT NULL,
    [ShipMethodID] [int] NOT NULL,
    [CreditCardID] [int] NULL,
    [CreditCardApprovalCode] [varchar](15) NULL,
    [CurrencyRateID] [int] NULL,
    [SubTotal] [money] NOT NULL,
    [TaxAmt] [money] NOT NULL ,
    [Freight] [money] NOT NULL ,
    [TotalDue]  AS (isnull(([SubTotal]+[TaxAmt])+[Freight],(0))),
    [Comment] [nvarchar](128) NULL,
    CONSTRAINT [PK_SalesOrderHeader_SalesOrderID] PRIMARY KEY CLUSTERED
    (
        [SalesOrderID] ASC
    )
)

CREATE TABLE [SalesOrderDetail](
    [SalesOrderID] [int] NOT NULL,
    [SalesOrderDetailID] [int] IDENTITY(1,1) NOT NULL,
    [CarrierTrackingNumber] [nvarchar](25) NULL,
    [OrderQty] [smallint] NOT NULL,
    [ProductID] [int] NOT NULL,
    [SpecialOfferID] [int] NOT NULL,
    [UnitPrice] [money] NOT NULL,
    [UnitPriceDiscount] [money],
    [LineTotal]  AS (isnull(([UnitPrice]*((1.0)-[UnitPriceDiscount]))*[OrderQty],(0.0))),
     CONSTRAINT [PK_SalesOrderDetail_SalesOrderID_SalesOrderDetailID] PRIMARY KEY CLUSTERED
    (
        [SalesOrderID] ASC,
        [SalesOrderDetailID] ASC
    )
)

From above tables, hubs, links and satellites are as below

  • Hubs (HProduct, HProductSubCategory, HProductCategory)
  • Links (LProducts, LOrderHeader, LOrderDetails)
  • Satellites (SHProudct, SHProductSubCategory, SHProductCategory, SLOrderHeader, SLOrderDetails)

Create Table [HProductCategory]
(
    HProductCategoryKey int identity(1,1) NOT NULL constraint  pk_hProductCateogry primary key clustered,
    ProductCategoryID int not null,
    LoadDate Timestamp,
    RecordSource varchar(100)
)

Create Table [SHProductCategory]
(
    HProductCategoryKey int not null,
    LoadDate datetime not null,
    [Name] nvarchar(100) not null
    constraint PK_SHproductCategory Primary Key Clustered
    (
        HProductCategoryKey,
        LoadDate
    )
)

Create Table HProductSubCategory(
HProductSubCategoryKey int identity(1,1) NOT NULL constraint PK_HProductSubCategory primary key clustered,
ProductSubCategoryID int NOT NULL,
LoadDate Timestamp,
RecordSource varchar(100)
)

Create Table SHProductSubCategory(
HProductSubCategoryKey int not null,
LoadDate TimeStamp not null,
Name varchar(100),
constraint [PK_ProductSubcategory] primary key clustered
    (
        HProductSubCategoryKey,
        LoadDate
    )
)

Create table HProduct
(
    HProductKey int identity(1,1) not null constraint PK_HProductKey primary key clustered,
    [ProductID] [int] IDENTITY(1,1) NOT NULL,
    LoadDate Timestamp,
    RecordSource varchar(100)

)

Create table SHProduct
(
    HProductKey int identity(1,1) not null,
    LoadDate datetime not null,
    [Name] nvarchar(100) NOT NULL,
    [ProductNumber] [nvarchar](25) NOT NULL,
    [Color] [nvarchar](15) NULL,
    [SafetyStockLevel] [smallint] NOT NULL,
    [ReorderPoint] [smallint] NOT NULL,
    [StandardCost] [money] NOT NULL,
    [ListPrice] [money] NOT NULL,
    [Size] [nvarchar](5) NULL,
    [SizeUnitMeasureCode] [nchar](3) NULL,
    [WeightUnitMeasureCode] [nchar](3) NULL,
    [Weight] [decimal](8, 2) NULL,
    [DaysToManufacture] [int] NOT NULL,
    [ProductLine] [nchar](2) NULL,
    [Class] [nchar](2) NULL,
    [Style] [nchar](2) NULL,
     [SellStartDate] [datetime] NOT NULL,
    [SellEndDate] [datetime] NULL,
    [DiscontinuedDate] [datetime] NULL,
    Constraint [PK_SHProductKey] Primary Key Clustered
    (
        HProductKey,
        LoadDate
    )
)

 

Create Table LProducts
(
    LProductsKey int identity(1,1) not null constraint PK_Products Primary key Clustered,
    HProductKey int not null,
    HProductSubCategoryKey int not null,
    HProductCategorykey int not null,
    LoadDate datetime not null,
    RecordSource datetime not null
)

Create table LOrderHeader
(
    LOrderHeaderKey int identity(1,1) not null constraint pk_LOrderHeaderKey Primary key Clustered,
    [SalesOrderID] int not null,
    [CustomerID] [int] NOT NULL,
    [SalesPersonID] [int] NULL,
    [TerritoryID] [int] NULL,
    [BillToAddressID] [int] NOT NULL,
    [ShipToAddressID] [int] NOT NULL,
    [ShipMethodID] [int] NOT NULL,
    [CreditCardID] [int] NULL,
    [CurrencyRateID] [int] NULL
)

Create table SLOrderHeader
(
    LOrderHeaderKey int not null,
    [OrderDate] [datetime] NOT NULL ,
    [SalesOrderNumber]  NOT NULL,
    [CreditCardApprovalCode] [varchar](15) NULL,
    [SubTotal] [money] NOT NULL,
    [TaxAmt] [money] NOT NULL ,
    [Freight] [money] NOT NULL ,
    [TotalDue]  AS (isnull(([SubTotal]+[TaxAmt])+[Freight],(0))),
    [Comment] [nvarchar](128) NULL
)

Create table LOrderDetail
(
    LOrderDetailKey int identity(1,1) constraint PK_OrderDetailKey primary key clustered,
    LOrderHeaderKey int not null,
    [SalesOrderDetailID] [int]  NOT NULL,
    LProductsKey int not null,
    [SpecialOfferID] [int] NOT NULL
)

Create table SLOrderDetail
(
    LOrderDetailKey int not null,
    [CarrierTrackingNumber] [nvarchar](25) NULL,
    [OrderQty] [smallint] NOT NULL,
    [UnitPrice] [money] NOT NULL,
    [UnitPriceDiscount] [money],
    [LineTotal]  AS (isnull(([UnitPrice]*((1.0)-[UnitPriceDiscount]))*[OrderQty],(0.0))),
)

This completes my first attempt to try to design a DV DWH model from 3NF of OLTP systems. But during design there are few questions that cropped up which for now are ignored and purpose is to address  / understand them in next couple of posts. Few of questions are below

  • If there is a parent and child relationship in tables,
    • In DV these need to be flattened in two ways
      • including parent attributes in Hubs or
      • create a link table that capture relationships between parent and child.
        • When creating do we need to create an additional link table or link table that is created for transactions will suffice?
          • Pros and cons of new linked vs leveraging existing link table.
  • An attribute is present in transaction table
    • What decides when to place attribute link or satellite tables.

Few of such questions are pondering that need to be learnt also I need to present above sql scripts in more presentable visual format.

Until my next learning

Abhyast