What's With These ASCII, EBCDIC, Unicode CCSIDs?

1 I want stress-free want want an System i 8 Copyright IBM Corporation, 2007. All Rights publication may refer to products that are not currently available in your country. IBM makes no commitment to make available any products referred to : What's with These ascii , ebcdic , Unicode CCSIDs? Bruce ViningSession: 51006125 CEIBM System i 2007 IBM Corporationi want an today's business world there is a growing need to exchange data with other users that might be working in different languages and environments. This might involve using Unicode to accept and display Russian and Japanese data from a 5250 RPG application, or general data that needs to be received or sent in batch to an AIX application.

2 This session covers how to use built-in facilities of i5/ OS to work with other systems using encodings such as ascii , ebcdic , and Unicode . Samples are provided in RPG, COBOL, C and the end of this session, attendees will be able to:1. Convert data using the iconv Support Unicode in a 5250 Support Unicode in a DB2 environment. IBM System i 2007 IBM Corporationi want an start with some terms character Set a collection of elements used to represent textual information ( 0-9, a-z, A-Z, .,;:!?/-_ @#$%^&*()+={}~` .. ) A character Set generally supports more than one language Latin-1 character Set supports all Western European languages Code Page (AKA Code set) where each character in a character set is assigned a numerical representation (often used interchangeably with character set charset in HTML) CCSID a unique number (0-65535) used by IBM to uniquely identify a Coded character Set and a Codepage.

3 IBM System i 2007 IBM Corporationi want an of an ebcdic code pageFixed CodePointsChangeableCode PointsExamples of Characters that do change hex values:#, $, @, IBM System i 2007 IBM Corporationi want an of a ascii code pageFixed CodePointsChangeableCode PointsIBM System i 2007 IBM Corporationi want an come so many different code pages in use?The codepage problem exists in both ascii and ebcdic ebcdic 10 different code pages to support Latin based script (English, French, German etc 37, 297, 500 etc 1 to support Greek (plus out of date ones) 1 to support Russian (plus out of date ones) etc ascii 2 code pages to support Latin based scripts 819 for ISO (8859-1) and 1252 for Windows 1 to support Greek (plus out of date ones) 1 to support Russian (plus out of date ones) etcIBM System i 2007 IBM Corporationi want an Considerations Coded character Set Identifiers (CCSIDs))

4 CCSIDs are used to define a method of assigning and preserving the meaning and rendering of characters through various stages of processing and interchange. CCSID support is particularly important when: Converting between encoding schemes ( ascii , ebcdic , Unicode ) Multiple national language versions, keyboards, and display stations are installed on i5/OS. Multiple System i servers are sharing data between systems with different national language versions. The correct keyboard support for a language is not available when you want to encode data in another language. i5/OS supports a large set of CCSIDs. i5/OS documents which pre-defined CCSID mappings it supports (which CCSIDs a given CCSID can be mapped to) Example: CCSID 00037 can be mapped to about 100 other CCSIDs Some CCSIDs only map to a few other CCSIDs.

5 To avoid needing to assign a CCSID to every object, set the CCSID at the system System i 2007 IBM Corporationi want an CCSID Values Defined on i5/OSCCSIDChar Set Description00037697US, Canada, Netherlands, Portugal, Brazil, New Zealand, Australia, others00256697 Netherlands00273697 Austria, Germany00277697 Denmark, Norway 00278697 Finland, Sweden00280697 Italy00284697 Spanish (Latin America)00285697 United Kingdom002901172 Japanese00297697 France009371175 Chinese Simplified010251150 Russian ..Note that the Western European languages share the same character SetIBM System i 2007 IBM Corporationi want an Integrity Problems Whenever data needs to be converted to a different CCSID and that CCSID has a different character set, the characters in the original CCSID data that do not exist in the destination CCSID will be replaced or substituted Enforced subset match Best fit Round trip Conversion is done character by character so not all characters in a field may be changed/lostIBM System i 2007 IBM Corporationi want an Example #1: Data integrity is not maintained Data integrity may not be maintained using CCSID 65535 across languages.

6 This CCSID is not recommended because it turns off automatic conversion. Example showing the purpose of maintaining data integrity. An application is being used by different language users. A database file created by a user contains a dollar sign and is read by a user in the United Kingdom and in Denmark. If the application does not assign CCSID tags that are associated with the data to the file, users see different characters. CountryKeyboard TypeCode page CCSIDCode point 5B $ 5B DenmarkDMB27765535X 5B IBM System i 2007 IBM Corporationi want an Example #2: Data integrity is maintained Data integrity is maintained by using CCSID tags. If the application assigns a CCSID associated with the data to a file, the application can use i5/OS CCSID support to maintain the integrity of the data.

7 When the file is created with CCSID 037, the user in the United Kingdom (job CCSID 285) and the user in Denmark (job CCSID 277) see the same character . Database management takes care of the mapping. CountryKeyboard TypeCode page CCSIDCode point 5B $ 4A $DenmarkDMB27700277X 67 $IBM System i 2007 IBM Corporationi want an what is Unicode ? Unicode is the universal character encoding standardused for representation of text for computer processing. Can be used to store & process all significant current & past languages Unicode provides a unique hex encoded number for every character , no matter what the platform, program or language The Unicode Standard has been adopted by industry leaders Apple, HP, IBM, Microsoft, Oracle, SAP, Sun, Sybase, Unisys many others.

8 Unicode is required by web users and modern standards XML, Java, ECMAS cript (JavaScript), LDAP, CORBA , WMLIBM System i 2007 IBM Corporationi want an Interactive Ship To DisplayIBM System i 2007 IBM Corporationi want an Interactive Ship To DisplayUsing English and CCSID 37 IBM System i 2007 IBM Corporationi want an Interactive Ship To Physical File DDSORDER (Order Summary):UNIQUE R ORDREC ORDNO 5 0 ORDSTS 1 COMPANY 40 CONTACT 40 ADDR1 40 ADDR2 40 K ORDNOINVEN (Inventory Descriptions):UNIQUE R INVREC PARTNO 5 0 PARTDESC 40 K PARTNOORDDET (Order Detail).

9 UNIQUE R ORDDEC ORDNO R REFFLD(ORDREC/ORDNO ORDER) PARTNO R REFFLD(INVREC/PARTNO INVEN) ORDERQTY 6 0 K ORDNO K PARTNO IBM System i 2007 IBM Corporationi want an Interactive Ship To Display File DDSCF03(03) * Command key prompts R FMT1 23 4'F3=Exit' * Prompt for Order Number R PROMPT 3 2'Order Number ..' ORDNO R I 3 28 REFFLD(ORDREC/ORDNO ORDER) 50 22 2'Incorrect Order Number' 23 4'F3=Exit' * Subfile for parts ordered R SFLRCD SFL PARTNO R O 12 4 REFFLD(ORDDEC/PARTNO ORDDET)EDTWRD(' , ') PARTDESC R O 12 12 REFFLD(INVREC/PARTDESC INVEN)ORDERQTY R O 12 65 EDTWRD(' , ') REFFLD(ORDDEC/ORDERQTY ORDDET) IBM System i 2007 IBM Corporationi want an Interactive Ship To Display File DDS* Subfile control and main display R SFLCTL SFLCTL(SFLRCD)

10 SFLSIZ(100) SFLPAG(9) N25 SFLDSPCTL OVERLAY 21 SFLDSP 25 SFLCLR 1 28'Ship To Information' 3 2'Company ..' COMPANY R O 3 28 REFFLD(ORDREC/COMPANY ORDER) 4 2'Contact ..' CONTACT R O 4 28 REFFLD(ORDREC/CONTACT ORDER) 6 2'Status ..' ORDSTS R O 6 28 REFFLD(ORDREC/ORDSTS ORDER) 8 2'Ship to address ..' ADDR1 R O 8 28 REFFLD(ORDREC/ADDR1 ORDER) ADDR2 R O 9 28 REFFLD(ORDREC/ADDR2 ORDER) 11 4'Part No' 11 12'Part Description' 11 65'Quantity' IBM System i 2007 IBM Corporationi want an ILE RPG Interactive ProgramFiles and Working Fieldsfshiptodspfcf e workstn sfile(sflrcd.)