analysis/analysis.txt

(venv) PS V:\Deeptanshu Lal\PROJECTS\Toxic Comment Classification> python .\analysis\analyze_lang_distribution.py
Reading dataset...

Dataset Overview:
--------------------------------------------------
Total number of comments: 361,228
Number of languages: 7

Language Distribution:
--------------------------------------------------
ru: 52,632 comments (14.57%)
tr: 52,558 comments (14.55%)
pt: 52,440 comments (14.52%)
es: 52,412 comments (14.51%)
fr: 52,368 comments (14.50%)
it: 52,340 comments (14.49%)
en: 46,478 comments (12.87%)

Class Distribution by Language:
--------------------------------------------------

RU (Total: 52,632 comments)
0 toxic classes: 26,316 (50.00%)
1 toxic classes: 7,688 (14.61%)
2 toxic classes: 8,010 (15.22%)
3 toxic classes: 7,103 (13.50%)
4 toxic classes: 2,740 (5.21%)
5 toxic classes: 706 (1.34%)
6 toxic classes: 69 (0.13%)

TR (Total: 52,558 comments)
0 toxic classes: 26,279 (50.00%)
1 toxic classes: 7,677 (14.61%)
2 toxic classes: 8,004 (15.23%)
3 toxic classes: 7,088 (13.49%)
4 toxic classes: 2,736 (5.21%)
5 toxic classes: 705 (1.34%)
6 toxic classes: 69 (0.13%)

PT (Total: 52,440 comments)
0 toxic classes: 26,220 (50.00%)
1 toxic classes: 7,668 (14.62%)
2 toxic classes: 7,977 (15.21%)
3 toxic classes: 7,071 (13.48%)
4 toxic classes: 2,732 (5.21%)
5 toxic classes: 703 (1.34%)
6 toxic classes: 69 (0.13%)

ES (Total: 52,412 comments)
0 toxic classes: 26,206 (50.00%)
1 toxic classes: 7,647 (14.59%)
2 toxic classes: 7,982 (15.23%)
3 toxic classes: 7,069 (13.49%)
4 toxic classes: 2,737 (5.22%)
5 toxic classes: 702 (1.34%)
6 toxic classes: 69 (0.13%)

FR (Total: 52,368 comments)
0 toxic classes: 26,184 (50.00%)
1 toxic classes: 7,626 (14.56%)
2 toxic classes: 7,990 (15.26%)
3 toxic classes: 7,066 (13.49%)
4 toxic classes: 2,728 (5.21%)
5 toxic classes: 705 (1.35%)
6 toxic classes: 69 (0.13%)

IT (Total: 52,340 comments)
0 toxic classes: 26,170 (50.00%)
1 toxic classes: 7,652 (14.62%)
2 toxic classes: 7,967 (15.22%)
3 toxic classes: 7,057 (13.48%)
4 toxic classes: 2,722 (5.20%)
5 toxic classes: 703 (1.34%)
6 toxic classes: 69 (0.13%)

EN (Total: 46,478 comments)
0 toxic classes: 22,989 (49.46%)
1 toxic classes: 8,499 (18.29%)
2 toxic classes: 5,604 (12.06%)
3 toxic classes: 6,391 (13.75%)
4 toxic classes: 2,395 (5.15%)
5 toxic classes: 553 (1.19%)
6 toxic classes: 47 (0.10%)

Detailed Toxicity Analysis by Language:
--------------------------------------------------

RU (Total: 52,632 comments)
- Toxic:
  Count: 25,954 (49.31%)
  95% CI: [48.89%, 49.74%]
- Severe Toxic:
  Count: 2,441 (4.64%)
  95% CI: [4.46%, 4.82%]
- Obscene:
  Count: 12,432 (23.62%)
  95% CI: [23.26%, 23.98%]
- Threat:
  Count: 1,075 (2.04%)
  95% CI: [1.92%, 2.16%]
- Insult:
  Count: 15,207 (28.89%)
  95% CI: [28.51%, 29.28%]
- Identity Hate:
  Count: 2,812 (5.34%)
  95% CI: [5.15%, 5.53%]

TR (Total: 52,558 comments)
- Toxic:
  Count: 25,908 (49.29%)
  95% CI: [48.87%, 49.72%]
- Severe Toxic:
  Count: 2,439 (4.64%)
  95% CI: [4.46%, 4.82%]
- Obscene:
  Count: 12,411 (23.61%)
  95% CI: [23.25%, 23.98%]
- Threat:
  Count: 1,077 (2.05%)
  95% CI: [1.93%, 2.17%]
- Insult:
  Count: 15,170 (28.86%)
  95% CI: [28.48%, 29.25%]
- Identity Hate:
  Count: 2,827 (5.38%)
  95% CI: [5.19%, 5.57%]

PT (Total: 52,440 comments)
- Toxic:
  Count: 25,841 (49.28%)
  95% CI: [48.85%, 49.71%]
- Severe Toxic:
  Count: 2,432 (4.64%)
  95% CI: [4.46%, 4.82%]
- Obscene:
  Count: 12,395 (23.64%)
  95% CI: [23.27%, 24.00%]
- Threat:
  Count: 1,080 (2.06%)
  95% CI: [1.94%, 2.18%]
- Insult:
  Count: 15,143 (28.88%)
  95% CI: [28.49%, 29.26%]
- Identity Hate:
  Count: 2,801 (5.34%)
  95% CI: [5.15%, 5.53%]

ES (Total: 52,412 comments)
- Toxic:
  Count: 25,874 (49.37%)
  95% CI: [48.94%, 49.79%]
- Severe Toxic:
  Count: 2,432 (4.64%)
  95% CI: [4.46%, 4.82%]
- Obscene:
  Count: 12,388 (23.64%)
  95% CI: [23.27%, 24.00%]
- Threat:
  Count: 1,073 (2.05%)
  95% CI: [1.93%, 2.17%]
- Insult:
  Count: 15,140 (28.89%)
  95% CI: [28.50%, 29.27%]
- Identity Hate:
  Count: 2,783 (5.31%)
  95% CI: [5.12%, 5.50%]

FR (Total: 52,368 comments)
- Toxic:
  Count: 25,877 (49.41%)
  95% CI: [48.99%, 49.84%]
- Severe Toxic:
  Count: 2,428 (4.64%)
  95% CI: [4.46%, 4.82%]
- Obscene:
  Count: 12,379 (23.64%)
  95% CI: [23.27%, 24.00%]
- Threat:
  Count: 1,066 (2.04%)
  95% CI: [1.91%, 2.16%]
- Insult:
  Count: 15,131 (28.89%)
  95% CI: [28.51%, 29.28%]
- Identity Hate:
  Count: 2,774 (5.30%)
  95% CI: [5.11%, 5.49%]

IT (Total: 52,340 comments)
- Toxic:
  Count: 25,827 (49.34%)
  95% CI: [48.92%, 49.77%]
- Severe Toxic:
  Count: 2,429 (4.64%)
  95% CI: [4.46%, 4.82%]
- Obscene:
  Count: 12,341 (23.58%)
  95% CI: [23.21%, 23.94%]
- Threat:
  Count: 1,077 (2.06%)
  95% CI: [1.94%, 2.18%]
- Insult:
  Count: 15,118 (28.88%)
  95% CI: [28.50%, 29.27%]
- Identity Hate:
  Count: 2,782 (5.32%)
  95% CI: [5.12%, 5.51%]

EN (Total: 46,478 comments)
- Toxic:
  Count: 22,343 (48.07%)
  95% CI: [47.62%, 48.53%]
- Severe Toxic:
  Count: 1,986 (4.27%)
  95% CI: [4.09%, 4.46%]
- Obscene:
  Count: 12,356 (26.58%)
  95% CI: [26.18%, 26.99%]
- Threat:
  Count: 1,204 (2.59%)
  95% CI: [2.45%, 2.73%]
- Insult:
  Count: 11,475 (24.69%)
  95% CI: [24.30%, 25.08%]
- Identity Hate:
  Count: 2,143 (4.61%)
  95% CI: [4.42%, 4.80%]

Statistical Analysis:
--------------------------------------------------

Chi-square test for number of toxic classes by language:
Chi-square statistic: 654.28
p-value: 0.0000000000
Significant at α=0.05: Yes

Chi-square test for Toxic:
Chi-square statistic: 26.10
p-value: 0.0002136602
Significant at α=0.05: Yes

Chi-square test for Severe Toxic:
Chi-square statistic: 12.38
p-value: 0.0540052211
Significant at α=0.05: No

Chi-square test for Obscene:
Chi-square statistic: 195.12
p-value: 0.0000000000
Significant at α=0.05: Yes

Chi-square test for Threat:
Chi-square statistic: 57.45
p-value: 0.0000000001
Significant at α=0.05: Yes

Chi-square test for Insult:
Chi-square statistic: 350.72
p-value: 0.0000000000
Significant at α=0.05: Yes

Chi-square test for Identity Hate:
Chi-square statistic: 42.77
p-value: 0.0000001295
Significant at α=0.05: Yes