Skip to Content (Press Enter)
2018-05-07 20-49-39
Mário Araújo
54Views
1Comments
Cast between Integer and Decimal data types (IntegerToDecimal)
Discussion
How-to
Hi,

Take a quick look at this assign: CResult = CountCorrects/GetAnswerQuestions.ReturnedRowCount*100
Where CountCorrects = 1 and GetAnswerQuestions.ReturnedRowCount = 1.

The result of this assign is 0 (even if I replace 100 with 100.0). Note that CResult is a decimal, all others are integers.

After some trial and error I found out that this expression CResult = IntegerToDecimal(CountCorrects)/IntegerToDecimal(GetAnswerQuestions.ReturnedRowCount)*100
has the behavior I was expecting (CResult =50).

The SS help mentions that there is an implicit cast between Integer and Decimal. What might be wrong here?
Cheers,
Mário
0
0
28 Aug 2007
Copy Link
2012-03-02 13-17-21
Paulo Ramos
Staff
Hi Mario,

(I believe you meant GetAnswerQuestions.ReturnedRowCount = 2, for resulting in 50.)

On the 1st case, you are performing a division operation between two integers. The assign (and the related cast) only happens later. So, you will lose precision at an intermediate step.

I guess what's being done 'under the hood' is more or less this, for the 1st scenario:

Temp1 (int) = V1 (int) / V2 (int) [integer division: 1 / 2 = 0]
Temp2 (int/dec) = Temp1 (int) * Lit (int/dec) [0 * 100 = 0]
CResult = Temp2

While on the 2nd scenario:

Temp1 (dec) = V1 (dec) / V2 (dec) [decimal division: 1 / 2 = 0.5]
Temp2 (dec) = Temp1 (dec) * Lit (int/dec) [0.5 * 100 = 50]
CResult = Temp2

By the way, if you modify scenario 1 by simply putting the multiplication upfront (CResult = 100 * Count / RowsCount), you'll get the correct result in this case (but could lose precision with other values).

And I'm pretty sure you'll get the very same results if you use VS/.Net.
0
0
28 Aug 2007
Copy Link
Community GuidelinesBe kind and respectful, give credit to the original source of content, and search for duplicates before posting.