Quantcast
Channel: Preserve Line Feeds and Carriage Returns Powershell - Stack Overflow
Viewing all articles
Browse latest Browse all 2

Preserve Line Feeds and Carriage Returns Powershell

$
0
0

I have a csv file originated from Linux Server, so they contains \n to mark the ending of a line. Now I process this file through a powershell script and replace it's 'text qualifier' to something else (that I do because I am using SSIS to upload the csv feed to database and for some odd reasons SSIS don't supports 'embedded text qualifiers')

Part of the script which do this replacement looks like this

gc $file.FullName |
    % { if($_.indexOf("|~|") -eq -1) {$_ -replace "`"((?:`"`"|.)*?)`"(?!`")", "|~|`$1|~|" -replace "`"`"", "`""} else {$_ -replace " ", " "}} |
    sc  $temppath

This scripts works fine but that also change the line feed at the end to \r\n That I understand should not have been that big of problem until I realized that my original feed also contains occasional \r in the description column which is also getting replaced with "\r\n" Now SSIS package is unable to identify where' the csv line ends.

I searched and found this is due to Get-Content which work line by line so I changed the command to following.

[System.IO.File]::ReadAllText($file.FullName) |
            % { if($_.indexOf("|~|") -eq -1) {$_ -replace "`"((?:`"`"|.)*?)`"(?!`")", "|~|`$1|~|" -replace "`"`"", "`""} else {$_ -replace " ", " "}} |
            sc  $temppath

That seems to solve my issue but now I am trapped with *"OutOfMemoryException" as some of the csv files are big (about 400-500 MB) *

Any Suggestions what I can possibily do? Perhaps some replacement for ReadAllText() that works for big files?


Viewing all articles
Browse latest Browse all 2

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>